dotCMS Server Load Issues

classic Classic list List threaded Threaded
43 messages Options
123
Reply | Threaded
Open this post in threaded view
|

dotCMS Server Load Issues

Joel Goodman
Hey all,

We've been having some strange server load issues with dotCMS 1.7 - wanted to put some feelers out and see if any of you have a)seen this and b) if an  upgrade might fix what we're seeing.

This has only happened a few times, but basically dotCMS will use up memory and keep that session open - today our load stats show 173 hours for one single session. We have 3 of 8 memory bays at 100% load, another at around 93%, and at that point our server stops responding and time out on client browser requests.

We're not sure what does this. We haven't changed any configuration, there's plenty of storage space. Memory requests just aren't being spread out across all system RAM.

Any help is super appreciated. It's annoying to have our site going down so often after 9 months of rock-solid service.

Joel

--
Joel Goodman ([hidden email]<mailto:[hidden email]>) || Web Content Coordinator
315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu>
Futuristic • Strategic • Responsibility • Restorative • Competition

Reply | Threaded
Open this post in threaded view
|

RE: dotCMS Server Load Issues

Todd Reed-2
What is your hardware and OS/Kernel version?    Typically the operating
system would keep track of the memory allocs.  Have you run any hardware
diags?

 

--Todd

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf
Of Joel Goodman
Sent: Friday, December 11, 2009 2:41 PM
To: [hidden email] [hidden email]
Subject: [dotcms] dotCMS Server Load Issues

 

 

Hey all,

 

We've been having some strange server load issues with dotCMS 1.7 -
wanted to put some feelers out and see if any of you have a)seen this
and b) if an  upgrade might fix what we're seeing.

 

This has only happened a few times, but basically dotCMS will use up
memory and keep that session open - today our load stats show 173 hours
for one single session. We have 3 of 8 memory bays at 100% load, another
at around 93%, and at that point our server stops responding and time
out on client browser requests.

 

We're not sure what does this. We haven't changed any configuration,
there's plenty of storage space. Memory requests just aren't being
spread out across all system RAM.

 

Any help is super appreciated. It's annoying to have our site going down
so often after 9 months of rock-solid service.

 

Joel


--

Joel Goodman ([hidden email]) || Web Content Coordinator

315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
www.greenville.edu

Futuristic * Strategic * Responsibility * Restorative * Competition

 



Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

Joel Goodman
Everything is spec'd from the dotCMS appliance recommendations (and then some) -

2.0 GHz Xeon Quad Core
8 GB RAM
4 x 300GB 15,000 RPM SAS drives in a RAID 10
Ubuntu 8.4 64-bit

Our server admin has been running checks. He's thought of checking the BIOS memory settings - but it's strange that everything should work fine until just the last few weeks.

Joel

--
Joel Goodman ([hidden email]<mailto:[hidden email]>) || Web Content Coordinator
315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu>
Futuristic • Strategic • Responsibility • Restorative • Competition

On Dec 11, 2009, at 3:05 PM, Todd Reed wrote:



What is your hardware and OS/Kernel version?    Typically the operating system would keep track of the memory allocs.  Have you run any hardware diags?

--Todd


From: [hidden email] [mailto:[hidden email]] On Behalf Of Joel Goodman
Sent: Friday, December 11, 2009 2:41 PM
To: [hidden email] [hidden email]
Subject: [dotcms] dotCMS Server Load Issues



Hey all,

We've been having some strange server load issues with dotCMS 1.7 - wanted to put some feelers out and see if any of you have a)seen this and b) if an  upgrade might fix what we're seeing.

This has only happened a few times, but basically dotCMS will use up memory and keep that session open - today our load stats show 173 hours for one single session. We have 3 of 8 memory bays at 100% load, another at around 93%, and at that point our server stops responding and time out on client browser requests.

We're not sure what does this. We haven't changed any configuration, there's plenty of storage space. Memory requests just aren't being spread out across all system RAM.

Any help is super appreciated. It's annoying to have our site going down so often after 9 months of rock-solid service.

Joel

--
Joel Goodman ([hidden email]<mailto:[hidden email]>) || Web Content Coordinator
315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu/>
Futuristic • Strategic • Responsibility • Restorative • Competition




Reply | Threaded
Open this post in threaded view
|

RE: dotCMS Server Load Issues

Todd Reed-2
What version of JavaJDK?  That is similar to our setup as far as OS,
etc.  

 

From: [hidden email] [mailto:[hidden email]] On Behalf
Of Joel Goodman
Sent: Friday, December 11, 2009 3:16 PM
To: [hidden email]
Subject: Re: [dotcms] dotCMS Server Load Issues

 

 

Everything is spec'd from the dotCMS appliance recommendations (and then
some) -

 

2.0 GHz Xeon Quad Core

8 GB RAM

4 x 300GB 15,000 RPM SAS drives in a RAID 10

Ubuntu 8.4 64-bit

 

Our server admin has been running checks. He's thought of checking the
BIOS memory settings - but it's strange that everything should work fine
until just the last few weeks.

 

Joel


--

Joel Goodman ([hidden email]) || Web Content Coordinator

315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
www.greenville.edu

Futuristic * Strategic * Responsibility * Restorative * Competition

 

On Dec 11, 2009, at 3:05 PM, Todd Reed wrote:





 

 

What is your hardware and OS/Kernel version?    Typically the operating
system would keep track of the memory allocs.  Have you run any hardware
diags?

 

--Todd

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf
Of Joel Goodman
Sent: Friday, December 11, 2009 2:41 PM
To: [hidden email] [hidden email]
Subject: [dotcms] dotCMS Server Load Issues

 

 

Hey all,

 

We've been having some strange server load issues with dotCMS 1.7 -
wanted to put some feelers out and see if any of you have a)seen this
and b) if an  upgrade might fix what we're seeing.

 

This has only happened a few times, but basically dotCMS will use up
memory and keep that session open - today our load stats show 173 hours
for one single session. We have 3 of 8 memory bays at 100% load, another
at around 93%, and at that point our server stops responding and time
out on client browser requests.

 

We're not sure what does this. We haven't changed any configuration,
there's plenty of storage space. Memory requests just aren't being
spread out across all system RAM.

 

Any help is super appreciated. It's annoying to have our site going down
so often after 9 months of rock-solid service.

 

Joel


--

Joel Goodman ([hidden email]) || Web Content Coordinator

315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
www.greenville.edu <http://www.greenville.edu/>

Futuristic * Strategic * Responsibility * Restorative * Competition

 

 

 



Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

Jim Serio-2
In reply to this post by Joel Goodman
Joel,

Are you also running Postgres?  If so, then you are experiencing the same
problem I have been for months.  The only way to resolve it, is to shut down
dotCMS, then restart Postgres (I stop, then start) and then start back up
dotCMS.

Jason looked over my logs and a process dump a few months back and deduced
there's something that creating some sort of SQL loop.  I have not been able
to reliably reproduce this but I can tell you that it has happend on the
following two occasions (since I've logged it):

1.) Someone searches for content.  The wheels will keep spinning and
Postgres will consume all CPU cycles bumping the system load (I've seen it
as high as 13!).

2.) Overwriting a file in the browser seems to cause the problem as well. It
happened with a few images.

Once the problem starts, the system load will never come back down until you
restart Postgres.

Of course, if you're not on Postgres, then disregard this :-)

Jim

On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman
<[hidden email]>wrote:

>
>
> Hey all,
>
> We've been having some strange server load issues with dotCMS 1.7 - wanted
> to put some feelers out and see if any of you have a)seen this and b) if an
>  upgrade might fix what we're seeing.
>
> This has only happened a few times, but basically dotCMS will use up memory
> and keep that session open - today our load stats show 173 hours for one
> single session. We have 3 of 8 memory bays at 100% load, another at around
> 93%, and at that point our server stops responding and time out on client
> browser requests.
>
> We're not sure what does this. We haven't changed any configuration,
> there's plenty of storage space. Memory requests just aren't being spread
> out across all system RAM.
>
> Any help is super appreciated. It's annoying to have our site going down so
> often after 9 months of rock-solid service.
>
> Joel
>
> --
> Joel Goodman ([hidden email]) || Web Content Coordinator
> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
> www.greenville.edu
> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

Joel Goodman
Yeah, we're running Postgres. Our Network Admin found that he has to do a physical machine reboot to get this to correct itself.

Has this been seen on any other db platforms? We've been contemplating a move to MySQL and maybe this will be the silver bullet.

I think our situation is really similar, if not the same judging by what we've observed.

Thanks for the help Jim!

Joel

--
Joel Goodman ([hidden email]<mailto:[hidden email]>) || Web Content Coordinator
315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu>
Futuristic • Strategic • Responsibility • Restorative • Competition

On Dec 11, 2009, at 4:37 PM, Jim Serio wrote:



Joel,

Are you also running Postgres?  If so, then you are experiencing the same problem I have been for months.  The only way to resolve it, is to shut down dotCMS, then restart Postgres (I stop, then start) and then start back up dotCMS.

Jason looked over my logs and a process dump a few months back and deduced there's something that creating some sort of SQL loop.  I have not been able to reliably reproduce this but I can tell you that it has happend on the following two occasions (since I've logged it):

1.) Someone searches for content.  The wheels will keep spinning and Postgres will consume all CPU cycles bumping the system load (I've seen it as high as 13!).

2.) Overwriting a file in the browser seems to cause the problem as well. It happened with a few images.

Once the problem starts, the system load will never come back down until you restart Postgres.

Of course, if you're not on Postgres, then disregard this :-)

Jim


On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman <[hidden email]<mailto:[hidden email]>> wrote:


Hey all,

We've been having some strange server load issues with dotCMS 1.7 - wanted to put some feelers out and see if any of you have a)seen this and b) if an  upgrade might fix what we're seeing.

This has only happened a few times, but basically dotCMS will use up memory and keep that session open - today our load stats show 173 hours for one single session. We have 3 of 8 memory bays at 100% load, another at around 93%, and at that point our server stops responding and time out on client browser requests.

We're not sure what does this. We haven't changed any configuration, there's plenty of storage space. Memory requests just aren't being spread out across all system RAM.

Any help is super appreciated. It's annoying to have our site going down so often after 9 months of rock-solid service.

Joel

--
Joel Goodman ([hidden email]<mailto:[hidden email]>) || Web Content Coordinator
315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu/>
Futuristic • Strategic • Responsibility • Restorative • Competition







Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

ajwagner777
Hey everyone,

We are having very similar problems and we are on MSSQL.  There is a theory among our group that this may be a memory leak, although we're not been able to get any internal resources to look into it.  Maybe Jason can spur the developers to start looking into this.

The problem is so bad that our system admin has written a script to restart the app every night at some late hour.  If it were not for this "preventative" measure, our system would be crashing on a bi-daily (sometimes daily) basis.

--- In [hidden email], Joel Goodman <joel.goodman@...> wrote:

>
> Yeah, we're running Postgres. Our Network Admin found that he has to do a physical machine reboot to get this to correct itself.
>
> Has this been seen on any other db platforms? We've been contemplating a move to MySQL and maybe this will be the silver bullet.
>
> I think our situation is really similar, if not the same judging by what we've observed.
>
> Thanks for the help Jim!
>
> Joel
>
> --
> Joel Goodman (Joel.Goodman@...<mailto:Joel.Goodman@...>) || Web Content Coordinator
> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu>
> Futuristic • Strategic • Responsibility • Restorative • Competition
>
> On Dec 11, 2009, at 4:37 PM, Jim Serio wrote:
>
>
>
> Joel,
>
> Are you also running Postgres?  If so, then you are experiencing the same problem I have been for months.  The only way to resolve it, is to shut down dotCMS, then restart Postgres (I stop, then start) and then start back up dotCMS.
>
> Jason looked over my logs and a process dump a few months back and deduced there's something that creating some sort of SQL loop.  I have not been able to reliably reproduce this but I can tell you that it has happend on the following two occasions (since I've logged it):
>
> 1.) Someone searches for content.  The wheels will keep spinning and Postgres will consume all CPU cycles bumping the system load (I've seen it as high as 13!).
>
> 2.) Overwriting a file in the browser seems to cause the problem as well. It happened with a few images.
>
> Once the problem starts, the system load will never come back down until you restart Postgres.
>
> Of course, if you're not on Postgres, then disregard this :-)
>
> Jim
>
>
> On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman <joel.goodman@...<mailto:joel.goodman@...>> wrote:
>
>
> Hey all,
>
> We've been having some strange server load issues with dotCMS 1.7 - wanted to put some feelers out and see if any of you have a)seen this and b) if an  upgrade might fix what we're seeing.
>
> This has only happened a few times, but basically dotCMS will use up memory and keep that session open - today our load stats show 173 hours for one single session. We have 3 of 8 memory bays at 100% load, another at around 93%, and at that point our server stops responding and time out on client browser requests.
>
> We're not sure what does this. We haven't changed any configuration, there's plenty of storage space. Memory requests just aren't being spread out across all system RAM.
>
> Any help is super appreciated. It's annoying to have our site going down so often after 9 months of rock-solid service.
>
> Joel
>
> --
> Joel Goodman (Joel.Goodman@...<mailto:Joel.Goodman@...>) || Web Content Coordinator
> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu/>
> Futuristic • Strategic • Responsibility • Restorative • Competition
>




------------------------------------

Need Help?  See:

http://www.dotcms.org/support 

for open source support options.Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/dotcms/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/dotcms/join
    (Yahoo! ID required)

<*> To change settings via email:
    [hidden email]
    [hidden email]

<*> To unsubscribe from this group, send an email to:
    [hidden email]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Reply | Threaded
Open this post in threaded view
|

Re: Re: dotCMS Server Load Issues

Michiel Rook-4
Hi,

> We are having very similar problems and we are on MSSQL.  There is a theory among our group that this may be a memory leak, although we're not been able to get any internal resources to look into it.  Maybe Jason can spur the developers to start looking into this.

Following this thread we interest, as we (Olaf and I) have been
experiencing similar problems, we are running MySQL as the db backend.

System seems fine for a while (the definition of 'while' varies from
restart to restart), then loads start spiking, we get Hibernate
exceptions in the log (slow queries) and out of heap errors, at which
point the instance is completely unresponsive.

regards,

--
Michiel Rook - Touchdown Consulting Services

E-mail:    [hidden email]
Website:   www.touchdownconsulting.nl
Telefoon:  06-18525049


This email and any attachments to it may be confidential and are
intended solely for the use of the individual to whom it is
addressed. If you are not the intended recipient of this email,
you must neither take any action based upon its contents, nor copy
or show it to anyone. Please contact the sender if you believe you
have received this email in error.


------------------------------------

Need Help?  See:

http://www.dotcms.org/support 

for open source support options.Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/dotcms/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/dotcms/join
    (Yahoo! ID required)

<*> To change settings via email:
    [hidden email]
    [hidden email]

<*> To unsubscribe from this group, send an email to:
    [hidden email]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

markleew
We have experienced this exact issue as well. We are using Oracle as our db. We restart our app every morning so it hasn't been as big of an issue, but it does still occur from time to time.

--- In [hidden email], Michiel Rook <michiel@...> wrote:

>
> Hi,
>
> > We are having very similar problems and we are on MSSQL.  There is a theory among our group that this may be a memory leak, although we're not been able to get any internal resources to look into it.  Maybe Jason can spur the developers to start looking into this.
>
> Following this thread we interest, as we (Olaf and I) have been
> experiencing similar problems, we are running MySQL as the db backend.
>
> System seems fine for a while (the definition of 'while' varies from
> restart to restart), then loads start spiking, we get Hibernate
> exceptions in the log (slow queries) and out of heap errors, at which
> point the instance is completely unresponsive.
>
> regards,
>
> --
> Michiel Rook - Touchdown Consulting Services
>
> E-mail:    info@...
> Website:   www.touchdownconsulting.nl
> Telefoon:  06-18525049
>
>
> This email and any attachments to it may be confidential and are
> intended solely for the use of the individual to whom it is
> addressed. If you are not the intended recipient of this email,
> you must neither take any action based upon its contents, nor copy
> or show it to anyone. Please contact the sender if you believe you
> have received this email in error.
>




------------------------------------

Need Help?  See:

http://www.dotcms.org/support 

for open source support options.Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/dotcms/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/dotcms/join
    (Yahoo! ID required)

<*> To change settings via email:
    [hidden email]
    [hidden email]

<*> To unsubscribe from this group, send an email to:
    [hidden email]

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/

Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

Jim Serio-2
In reply to this post by Joel Goodman
Joel,

You admin should not have to restart the server - unless he doesn't know how
to restart the Postgres daemon :-)  The next time this happens, ask him to
run 'top' and see if the 'postmaster' process is the culprit.  If it is,
then our problems are related and perhaps someone at dotCMS can  Jason or
someone can increase their efforts on this.

Regards,
   Jim

On Mon, Dec 14, 2009 at 7:13 AM, Joel Goodman
<[hidden email]>wrote:

>
>
> Yeah, we're running Postgres. Our Network Admin found that he has to do a
> physical machine reboot to get this to correct itself.
>
> Has this been seen on any other db platforms? We've been contemplating a
> move to MySQL and maybe this will be the silver bullet.
>
> I think our situation is really similar, if not the same judging by what
> we've observed.
>
> Thanks for the help Jim!
>
> Joel
>
> --
> Joel Goodman ([hidden email]) || Web Content Coordinator
> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
> www.greenville.edu
> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>
> On Dec 11, 2009, at 4:37 PM, Jim Serio wrote:
>
>
>
> Joel,
>
> Are you also running Postgres?  If so, then you are experiencing the same
> problem I have been for months.  The only way to resolve it, is to shut down
> dotCMS, then restart Postgres (I stop, then start) and then start back up
> dotCMS.
>
> Jason looked over my logs and a process dump a few months back and deduced
> there's something that creating some sort of SQL loop.  I have not been able
> to reliably reproduce this but I can tell you that it has happend on the
> following two occasions (since I've logged it):
>
> 1.) Someone searches for content.  The wheels will keep spinning and
> Postgres will consume all CPU cycles bumping the system load (I've seen it
> as high as 13!).
>
> 2.) Overwriting a file in the browser seems to cause the problem as well.
> It happened with a few images.
>
> Once the problem starts, the system load will never come back down until
> you restart Postgres.
>
> Of course, if you're not on Postgres, then disregard this :-)
>
> Jim
>
> On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman <[hidden email]
> > wrote:
>
>>
>>
>> Hey all,
>>
>> We've been having some strange server load issues with dotCMS 1.7 - wanted
>> to put some feelers out and see if any of you have a)seen this and b) if an
>>  upgrade might fix what we're seeing.
>>
>> This has only happened a few times, but basically dotCMS will use up
>> memory and keep that session open - today our load stats show 173 hours for
>> one single session. We have 3 of 8 memory bays at 100% load, another at
>> around 93%, and at that point our server stops responding and time out on
>> client browser requests.
>>
>> We're not sure what does this. We haven't changed any configuration,
>> there's plenty of storage space. Memory requests just aren't being spread
>> out across all system RAM.
>>
>> Any help is super appreciated. It's annoying to have our site going down
>> so often after 9 months of rock-solid service.
>>
>> Joel
>>
>> --
>> Joel Goodman ([hidden email]) || Web Content Coordinator
>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>> www.greenville.edu
>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>
>>
>>
>>
>
>
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

Joel Goodman
I think he was restarting postgres and dotcms in reverse order and not getting the results he expected. I've sent this discussion on to him and we're going to do some digging before we all leave for Christmas.

Thanks for everyone's help and experiences!

Joel

--
Joel Goodman ([hidden email]<mailto:[hidden email]>) || Web Content Coordinator
315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu>
Futuristic • Strategic • Responsibility • Restorative • Competition

On Dec 14, 2009, at 10:38 AM, Jim Serio wrote:



Joel,

You admin should not have to restart the server - unless he doesn't know how to restart the Postgres daemon :-)  The next time this happens, ask him to run 'top' and see if the 'postmaster' process is the culprit.  If it is, then our problems are related and perhaps someone at dotCMS can  Jason or someone can increase their efforts on this.

Regards,
   Jim


On Mon, Dec 14, 2009 at 7:13 AM, Joel Goodman <[hidden email]<mailto:[hidden email]>> wrote:


Yeah, we're running Postgres. Our Network Admin found that he has to do a physical machine reboot to get this to correct itself.

Has this been seen on any other db platforms? We've been contemplating a move to MySQL and maybe this will be the silver bullet.

I think our situation is really similar, if not the same judging by what we've observed.

Thanks for the help Jim!

Joel

--
Joel Goodman ([hidden email]<mailto:[hidden email]>) || Web Content Coordinator
315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu/>
Futuristic • Strategic • Responsibility • Restorative • Competition

On Dec 11, 2009, at 4:37 PM, Jim Serio wrote:



Joel,

Are you also running Postgres?  If so, then you are experiencing the same problem I have been for months.  The only way to resolve it, is to shut down dotCMS, then restart Postgres (I stop, then start) and then start back up dotCMS.

Jason looked over my logs and a process dump a few months back and deduced there's something that creating some sort of SQL loop.  I have not been able to reliably reproduce this but I can tell you that it has happend on the following two occasions (since I've logged it):

1.) Someone searches for content.  The wheels will keep spinning and Postgres will consume all CPU cycles bumping the system load (I've seen it as high as 13!).

2.) Overwriting a file in the browser seems to cause the problem as well. It happened with a few images.

Once the problem starts, the system load will never come back down until you restart Postgres.

Of course, if you're not on Postgres, then disregard this :-)

Jim


On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman <[hidden email]<mailto:[hidden email]>> wrote:


Hey all,

We've been having some strange server load issues with dotCMS 1.7 - wanted to put some feelers out and see if any of you have a)seen this and b) if an  upgrade might fix what we're seeing.

This has only happened a few times, but basically dotCMS will use up memory and keep that session open - today our load stats show 173 hours for one single session. We have 3 of 8 memory bays at 100% load, another at around 93%, and at that point our server stops responding and time out on client browser requests.

We're not sure what does this. We haven't changed any configuration, there's plenty of storage space. Memory requests just aren't being spread out across all system RAM.

Any help is super appreciated. It's annoying to have our site going down so often after 9 months of rock-solid service.

Joel

--
Joel Goodman ([hidden email]<mailto:[hidden email]>) || Web Content Coordinator
315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu/>
Futuristic • Strategic • Responsibility • Restorative • Competition












Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

Jim Serio-2
I always worry about the order of shutting things down.  Long ago I had to
shut down dotCMS and I suspect this somehow corrupted data inside Postgres
since afterward our Tasks have never worked.  What I've done is shut down
dotCMS, then Postgres which seems logical but who knows.

Jim

On Mon, Dec 14, 2009 at 9:53 AM, Joel Goodman
<[hidden email]>wrote:

>
>
> I think he was restarting postgres and dotcms in reverse order and not
> getting the results he expected. I've sent this discussion on to him and
> we're going to do some digging before we all leave for Christmas.
>
> Thanks for everyone's help and experiences!
>
> Joel
>
> --
> Joel Goodman ([hidden email]) || Web Content Coordinator
> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
> www.greenville.edu
> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>
> On Dec 14, 2009, at 10:38 AM, Jim Serio wrote:
>
>
>
> Joel,
>
> You admin should not have to restart the server - unless he doesn't know
> how to restart the Postgres daemon :-)  The next time this happens, ask him
> to run 'top' and see if the 'postmaster' process is the culprit.  If it is,
> then our problems are related and perhaps someone at dotCMS can  Jason or
> someone can increase their efforts on this.
>
> Regards,
>    Jim
>
> On Mon, Dec 14, 2009 at 7:13 AM, Joel Goodman <[hidden email]
> > wrote:
>
>>
>>
>> Yeah, we're running Postgres. Our Network Admin found that he has to do a
>> physical machine reboot to get this to correct itself.
>>
>> Has this been seen on any other db platforms? We've been contemplating a
>> move to MySQL and maybe this will be the silver bullet.
>>
>> I think our situation is really similar, if not the same judging by what
>> we've observed.
>>
>> Thanks for the help Jim!
>>
>> Joel
>>
>> --
>> Joel Goodman ([hidden email]) || Web Content Coordinator
>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>> www.greenville.edu
>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>
>> On Dec 11, 2009, at 4:37 PM, Jim Serio wrote:
>>
>>
>>
>> Joel,
>>
>> Are you also running Postgres?  If so, then you are experiencing the same
>> problem I have been for months.  The only way to resolve it, is to shut down
>> dotCMS, then restart Postgres (I stop, then start) and then start back up
>> dotCMS.
>>
>> Jason looked over my logs and a process dump a few months back and deduced
>> there's something that creating some sort of SQL loop.  I have not been able
>> to reliably reproduce this but I can tell you that it has happend on the
>> following two occasions (since I've logged it):
>>
>> 1.) Someone searches for content.  The wheels will keep spinning and
>> Postgres will consume all CPU cycles bumping the system load (I've seen it
>> as high as 13!).
>>
>> 2.) Overwriting a file in the browser seems to cause the problem as well.
>> It happened with a few images.
>>
>> Once the problem starts, the system load will never come back down until
>> you restart Postgres.
>>
>> Of course, if you're not on Postgres, then disregard this :-)
>>
>> Jim
>>
>> On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman <
>> [hidden email]> wrote:
>>
>>>
>>>
>>> Hey all,
>>>
>>> We've been having some strange server load issues with dotCMS 1.7 -
>>> wanted to put some feelers out and see if any of you have a)seen this and b)
>>> if an  upgrade might fix what we're seeing.
>>>
>>> This has only happened a few times, but basically dotCMS will use up
>>> memory and keep that session open - today our load stats show 173 hours for
>>> one single session. We have 3 of 8 memory bays at 100% load, another at
>>> around 93%, and at that point our server stops responding and time out on
>>> client browser requests.
>>>
>>> We're not sure what does this. We haven't changed any configuration,
>>> there's plenty of storage space. Memory requests just aren't being spread
>>> out across all system RAM.
>>>
>>>  Any help is super appreciated. It's annoying to have our site going
>>> down so often after 9 months of rock-solid service.
>>>
>>> Joel
>>>
>>> --
>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>> www.greenville.edu
>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>
>
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

Joel Goodman
Jim, our net admin said he ran htop and said that postmaster was not the culprit...

Joel

--
Joel Goodman ([hidden email]<mailto:[hidden email]>) || Web Content Coordinator
315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu>
Futuristic • Strategic • Responsibility • Restorative • Competition

On Dec 14, 2009, at 11:13 AM, Jim Serio wrote:



I always worry about the order of shutting things down.  Long ago I had to shut down dotCMS and I suspect this somehow corrupted data inside Postgres since afterward our Tasks have never worked.  What I've done is shut down dotCMS, then Postgres which seems logical but who knows.

Jim


On Mon, Dec 14, 2009 at 9:53 AM, Joel Goodman <[hidden email]<mailto:[hidden email]>> wrote:


I think he was restarting postgres and dotcms in reverse order and not getting the results he expected. I've sent this discussion on to him and we're going to do some digging before we all leave for Christmas.

Thanks for everyone's help and experiences!

Joel

--
Joel Goodman ([hidden email]<mailto:[hidden email]>) || Web Content Coordinator
315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu/>
Futuristic • Strategic • Responsibility • Restorative • Competition

On Dec 14, 2009, at 10:38 AM, Jim Serio wrote:



Joel,

You admin should not have to restart the server - unless he doesn't know how to restart the Postgres daemon :-)  The next time this happens, ask him to run 'top' and see if the 'postmaster' process is the culprit.  If it is, then our problems are related and perhaps someone at dotCMS can  Jason or someone can increase their efforts on this.

Regards,
   Jim


On Mon, Dec 14, 2009 at 7:13 AM, Joel Goodman <[hidden email]<mailto:[hidden email]>> wrote:


Yeah, we're running Postgres. Our Network Admin found that he has to do a physical machine reboot to get this to correct itself.

Has this been seen on any other db platforms? We've been contemplating a move to MySQL and maybe this will be the silver bullet.

I think our situation is really similar, if not the same judging by what we've observed.

Thanks for the help Jim!

Joel

--
Joel Goodman ([hidden email]<mailto:[hidden email]>) || Web Content Coordinator
315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu/>
Futuristic • Strategic • Responsibility • Restorative • Competition

On Dec 11, 2009, at 4:37 PM, Jim Serio wrote:



Joel,

Are you also running Postgres?  If so, then you are experiencing the same problem I have been for months.  The only way to resolve it, is to shut down dotCMS, then restart Postgres (I stop, then start) and then start back up dotCMS.

Jason looked over my logs and a process dump a few months back and deduced there's something that creating some sort of SQL loop.  I have not been able to reliably reproduce this but I can tell you that it has happend on the following two occasions (since I've logged it):

1.) Someone searches for content.  The wheels will keep spinning and Postgres will consume all CPU cycles bumping the system load (I've seen it as high as 13!).

2.) Overwriting a file in the browser seems to cause the problem as well. It happened with a few images.

Once the problem starts, the system load will never come back down until you restart Postgres.

Of course, if you're not on Postgres, then disregard this :-)

Jim


On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman <[hidden email]<mailto:[hidden email]>> wrote:


Hey all,

We've been having some strange server load issues with dotCMS 1.7 - wanted to put some feelers out and see if any of you have a)seen this and b) if an  upgrade might fix what we're seeing.

This has only happened a few times, but basically dotCMS will use up memory and keep that session open - today our load stats show 173 hours for one single session. We have 3 of 8 memory bays at 100% load, another at around 93%, and at that point our server stops responding and time out on client browser requests.

We're not sure what does this. We haven't changed any configuration, there's plenty of storage space. Memory requests just aren't being spread out across all system RAM.

Any help is super appreciated. It's annoying to have our site going down so often after 9 months of rock-solid service.

Joel

--
Joel Goodman ([hidden email]<mailto:[hidden email]>) || Web Content Coordinator
315 E College Ave | Greenville, IL 62246 | (618) 664-6515 | www.greenville.edu<http://www.greenville.edu/>
Futuristic • Strategic • Responsibility • Restorative • Competition

















Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

olafmolilink
Hello,

tonight we're living in a world of dotCMS pain. Our dotCMS production server
decided to go bang on us, getting unresponsive and taking all the resources
with it after several minutes...killing and restarting dotCMS and other
daemons repeats the same problem over and over. Thanks to some deep
investigations of Michiel, who's working with us on this project, we found
out that the old-generation heap seams to cause our problem. It fills up
real quick after startup to 100% and then the full GC kicks in. The full GC
keeps on running and claims all the resources, shutting everything down. As
is described here:

http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp

"If there is a memory leak or inadequate heap allocated, eventually the
older generation will start to run out of room causing the Full GC thread to
run (nearly) continuously. Since this process "stops the world", Resin won't
be able to respond to requests and they'll start to back up. "

Anybody any idea what to do? We're already experimenting with the
heap-settings, we disabled the spellchecker, the database (mySQL) is stable
and tuned allright, there is enough memory available. We're out of options
right now...

Olaf



On Mon, Dec 14, 2009 at 6:59 PM, Joel Goodman
<[hidden email]>wrote:

>
>
> Jim, our net admin said he ran htop and said that postmaster was not the
> culprit...
>
> Joel
>
> --
> Joel Goodman ([hidden email]) || Web Content Coordinator
> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
> www.greenville.edu
> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>
> On Dec 14, 2009, at 11:13 AM, Jim Serio wrote:
>
>
>
> I always worry about the order of shutting things down.  Long ago I had to
> shut down dotCMS and I suspect this somehow corrupted data inside Postgres
> since afterward our Tasks have never worked.  What I've done is shut down
> dotCMS, then Postgres which seems logical but who knows.
>
> Jim
>
> On Mon, Dec 14, 2009 at 9:53 AM, Joel Goodman <[hidden email]
> > wrote:
>
>>
>>
>> I think he was restarting postgres and dotcms in reverse order and not
>> getting the results he expected. I've sent this discussion on to him and
>> we're going to do some digging before we all leave for Christmas.
>>
>> Thanks for everyone's help and experiences!
>>
>> Joel
>>
>> --
>> Joel Goodman ([hidden email]) || Web Content Coordinator
>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>> www.greenville.edu
>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>
>> On Dec 14, 2009, at 10:38 AM, Jim Serio wrote:
>>
>>
>>
>> Joel,
>>
>> You admin should not have to restart the server - unless he doesn't know
>> how to restart the Postgres daemon :-)  The next time this happens, ask him
>> to run 'top' and see if the 'postmaster' process is the culprit.  If it is,
>> then our problems are related and perhaps someone at dotCMS can  Jason or
>> someone can increase their efforts on this.
>>
>> Regards,
>>    Jim
>>
>> On Mon, Dec 14, 2009 at 7:13 AM, Joel Goodman <
>> [hidden email]> wrote:
>>
>>>
>>>
>>> Yeah, we're running Postgres. Our Network Admin found that he has to do a
>>> physical machine reboot to get this to correct itself.
>>>
>>> Has this been seen on any other db platforms? We've been contemplating a
>>> move to MySQL and maybe this will be the silver bullet.
>>>
>>> I think our situation is really similar, if not the same judging by what
>>> we've observed.
>>>
>>> Thanks for the help Jim!
>>>
>>> Joel
>>>
>>> --
>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>> www.greenville.edu
>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>
>>> On Dec 11, 2009, at 4:37 PM, Jim Serio wrote:
>>>
>>>
>>>
>>> Joel,
>>>
>>> Are you also running Postgres?  If so, then you are experiencing the same
>>> problem I have been for months.  The only way to resolve it, is to shut down
>>> dotCMS, then restart Postgres (I stop, then start) and then start back up
>>> dotCMS.
>>>
>>> Jason looked over my logs and a process dump a few months back and
>>> deduced there's something that creating some sort of SQL loop.  I have not
>>> been able to reliably reproduce this but I can tell you that it has happend
>>> on the following two occasions (since I've logged it):
>>>
>>> 1.) Someone searches for content.  The wheels will keep spinning and
>>> Postgres will consume all CPU cycles bumping the system load (I've seen it
>>> as high as 13!).
>>>
>>> 2.) Overwriting a file in the browser seems to cause the problem as well.
>>> It happened with a few images.
>>>
>>> Once the problem starts, the system load will never come back down until
>>> you restart Postgres.
>>>
>>> Of course, if you're not on Postgres, then disregard this :-)
>>>
>>> Jim
>>>
>>> On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman <
>>> [hidden email]> wrote:
>>>
>>>>
>>>>
>>>> Hey all,
>>>>
>>>> We've been having some strange server load issues with dotCMS 1.7 -
>>>> wanted to put some feelers out and see if any of you have a)seen this and b)
>>>> if an  upgrade might fix what we're seeing.
>>>>
>>>> This has only happened a few times, but basically dotCMS will use up
>>>> memory and keep that session open - today our load stats show 173 hours for
>>>> one single session. We have 3 of 8 memory bays at 100% load, another at
>>>> around 93%, and at that point our server stops responding and time out on
>>>> client browser requests.
>>>>
>>>> We're not sure what does this. We haven't changed any configuration,
>>>> there's plenty of storage space. Memory requests just aren't being spread
>>>> out across all system RAM.
>>>>
>>>>  Any help is super appreciated. It's annoying to have our site going
>>>> down so often after 9 months of rock-solid service.
>>>>
>>>> Joel
>>>>
>>>> --
>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>> www.greenville.edu
>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>
>
>  
>



--

Met vriendelijke groet / Kind regards,

Olaf Molenveld
----------------------------------------------
Interactive Link
Keizerstraat 31
3512 EA Utrecht
T +31 (0)30 634 32 66
F +31 (0)30 634 32 65
www.interactivelink.nl
[hidden email]

INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
PARODOS DEEL UIT VAN DE HANSNEL GROEP
----------------------------------------------
De informatie in dit e-mail bericht is uitsluitend bestemd voor de
geadresseerde.
Verstrekking aan en gebruik door anderen is niet toegestaan.
Door de elektronische verzending van het bericht kunnen er geen rechten
worden ontleend aan de informatie.
Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

Jim Serio-2
Olaf,

You have Michael working on the site?  Were any changes made to it right
before the problem arose?  Perhaps any changes made can be reverted to see
if they are the cause?

Jim

On Mon, Dec 14, 2009 at 3:01 PM, Olaf iLink <[hidden email]> wrote:

>
>
> Hello,
>
> tonight we're living in a world of dotCMS pain. Our dotCMS production
> server decided to go bang on us, getting unresponsive and taking all the
> resources with it after several minutes...killing and restarting dotCMS and
> other daemons repeats the same problem over and over. Thanks to some deep
> investigations of Michiel, who's working with us on this project, we found
> out that the old-generation heap seams to cause our problem. It fills up
> real quick after startup to 100% and then the full GC kicks in. The full GC
> keeps on running and claims all the resources, shutting everything down. As
> is described here:
>
> http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp
>
> "If there is a memory leak or inadequate heap allocated, eventually the
> older generation will start to run out of room causing the Full GC thread to
> run (nearly) continuously. Since this process "stops the world", Resin won't
> be able to respond to requests and they'll start to back up. "
>
> Anybody any idea what to do? We're already experimenting with the
> heap-settings, we disabled the spellchecker, the database (mySQL) is stable
> and tuned allright, there is enough memory available. We're out of options
> right now...
>
> Olaf
>
>
>
>
> On Mon, Dec 14, 2009 at 6:59 PM, Joel Goodman <[hidden email]
> > wrote:
>
>>
>>
>> Jim, our net admin said he ran htop and said that postmaster was not the
>> culprit...
>>
>> Joel
>>
>> --
>> Joel Goodman ([hidden email]) || Web Content Coordinator
>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>> www.greenville.edu
>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>
>> On Dec 14, 2009, at 11:13 AM, Jim Serio wrote:
>>
>>
>>
>> I always worry about the order of shutting things down.  Long ago I had to
>> shut down dotCMS and I suspect this somehow corrupted data inside Postgres
>> since afterward our Tasks have never worked.  What I've done is shut down
>> dotCMS, then Postgres which seems logical but who knows.
>>
>> Jim
>>
>> On Mon, Dec 14, 2009 at 9:53 AM, Joel Goodman <
>> [hidden email]> wrote:
>>
>>>
>>>
>>> I think he was restarting postgres and dotcms in reverse order and not
>>> getting the results he expected. I've sent this discussion on to him and
>>> we're going to do some digging before we all leave for Christmas.
>>>
>>> Thanks for everyone's help and experiences!
>>>
>>> Joel
>>>
>>> --
>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>> www.greenville.edu
>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>
>>> On Dec 14, 2009, at 10:38 AM, Jim Serio wrote:
>>>
>>>
>>>
>>> Joel,
>>>
>>> You admin should not have to restart the server - unless he doesn't know
>>> how to restart the Postgres daemon :-)  The next time this happens, ask him
>>> to run 'top' and see if the 'postmaster' process is the culprit.  If it is,
>>> then our problems are related and perhaps someone at dotCMS can  Jason or
>>> someone can increase their efforts on this.
>>>
>>> Regards,
>>>    Jim
>>>
>>> On Mon, Dec 14, 2009 at 7:13 AM, Joel Goodman <
>>> [hidden email]> wrote:
>>>
>>>>
>>>>
>>>> Yeah, we're running Postgres. Our Network Admin found that he has to do
>>>> a physical machine reboot to get this to correct itself.
>>>>
>>>> Has this been seen on any other db platforms? We've been contemplating a
>>>> move to MySQL and maybe this will be the silver bullet.
>>>>
>>>> I think our situation is really similar, if not the same judging by what
>>>> we've observed.
>>>>
>>>> Thanks for the help Jim!
>>>>
>>>> Joel
>>>>
>>>> --
>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>> www.greenville.edu
>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>
>>>> On Dec 11, 2009, at 4:37 PM, Jim Serio wrote:
>>>>
>>>>
>>>>
>>>> Joel,
>>>>
>>>> Are you also running Postgres?  If so, then you are experiencing the
>>>> same problem I have been for months.  The only way to resolve it, is to shut
>>>> down dotCMS, then restart Postgres (I stop, then start) and then start back
>>>> up dotCMS.
>>>>
>>>> Jason looked over my logs and a process dump a few months back and
>>>> deduced there's something that creating some sort of SQL loop.  I have not
>>>> been able to reliably reproduce this but I can tell you that it has happend
>>>> on the following two occasions (since I've logged it):
>>>>
>>>> 1.) Someone searches for content.  The wheels will keep spinning and
>>>> Postgres will consume all CPU cycles bumping the system load (I've seen it
>>>> as high as 13!).
>>>>
>>>> 2.) Overwriting a file in the browser seems to cause the problem as
>>>> well. It happened with a few images.
>>>>
>>>> Once the problem starts, the system load will never come back down until
>>>> you restart Postgres.
>>>>
>>>> Of course, if you're not on Postgres, then disregard this :-)
>>>>
>>>> Jim
>>>>
>>>> On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman <
>>>> [hidden email]> wrote:
>>>>
>>>>>
>>>>>
>>>>> Hey all,
>>>>>
>>>>> We've been having some strange server load issues with dotCMS 1.7 -
>>>>> wanted to put some feelers out and see if any of you have a)seen this and b)
>>>>> if an  upgrade might fix what we're seeing.
>>>>>
>>>>> This has only happened a few times, but basically dotCMS will use up
>>>>> memory and keep that session open - today our load stats show 173 hours for
>>>>> one single session. We have 3 of 8 memory bays at 100% load, another at
>>>>> around 93%, and at that point our server stops responding and time out on
>>>>> client browser requests.
>>>>>
>>>>> We're not sure what does this. We haven't changed any configuration,
>>>>> there's plenty of storage space. Memory requests just aren't being spread
>>>>> out across all system RAM.
>>>>>
>>>>>  Any help is super appreciated. It's annoying to have our site going
>>>>> down so often after 9 months of rock-solid service.
>>>>>
>>>>> Joel
>>>>>
>>>>> --
>>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>> www.greenville.edu
>>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
> --
>
> Met vriendelijke groet / Kind regards,
>
> Olaf Molenveld
> ----------------------------------------------
> Interactive Link
> Keizerstraat 31
> 3512 EA Utrecht
> T +31 (0)30 634 32 66
> F +31 (0)30 634 32 65
> www.interactivelink.nl
> [hidden email]
>
> INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
> PARODOS DEEL UIT VAN DE HANSNEL GROEP
> ----------------------------------------------
> De informatie in dit e-mail bericht is uitsluitend bestemd voor de
> geadresseerde.
> Verstrekking aan en gebruik door anderen is niet toegestaan.
> Door de elektronische verzending van het bericht kunnen er geen rechten
> worden ontleend aan de informatie.
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

olafmolilink
Hello Jim,

nope, we didn't change anything, the problem seems to appear spontaneously.
We already pruned the versioning, re-indexed the content, flushed all caches
etc.etc. but still the same behaviour..

Olaf

On Mon, Dec 14, 2009 at 11:20 PM, Jim Serio <[hidden email]> wrote:

>
>
> Olaf,
>
> You have Michael working on the site?  Were any changes made to it right
> before the problem arose?  Perhaps any changes made can be reverted to see
> if they are the cause?
>
> Jim
>
> On Mon, Dec 14, 2009 at 3:01 PM, Olaf iLink <[hidden email]> wrote:
>
>>
>>
>> Hello,
>>
>> tonight we're living in a world of dotCMS pain. Our dotCMS production
>> server decided to go bang on us, getting unresponsive and taking all the
>> resources with it after several minutes...killing and restarting dotCMS and
>> other daemons repeats the same problem over and over. Thanks to some deep
>> investigations of Michiel, who's working with us on this project, we found
>> out that the old-generation heap seams to cause our problem. It fills up
>> real quick after startup to 100% and then the full GC kicks in. The full GC
>> keeps on running and claims all the resources, shutting everything down. As
>> is described here:
>>
>> http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp
>>
>> "If there is a memory leak or inadequate heap allocated, eventually the
>> older generation will start to run out of room causing the Full GC thread to
>> run (nearly) continuously. Since this process "stops the world", Resin won't
>> be able to respond to requests and they'll start to back up. "
>>
>> Anybody any idea what to do? We're already experimenting with the
>> heap-settings, we disabled the spellchecker, the database (mySQL) is stable
>> and tuned allright, there is enough memory available. We're out of options
>> right now...
>>
>> Olaf
>>
>>
>>
>>
>> On Mon, Dec 14, 2009 at 6:59 PM, Joel Goodman <
>> [hidden email]> wrote:
>>
>>>
>>>
>>> Jim, our net admin said he ran htop and said that postmaster was not the
>>> culprit...
>>>
>>> Joel
>>>
>>> --
>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>> www.greenville.edu
>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>
>>> On Dec 14, 2009, at 11:13 AM, Jim Serio wrote:
>>>
>>>
>>>
>>> I always worry about the order of shutting things down.  Long ago I had
>>> to shut down dotCMS and I suspect this somehow corrupted data inside
>>> Postgres since afterward our Tasks have never worked.  What I've done is
>>> shut down dotCMS, then Postgres which seems logical but who knows.
>>>
>>> Jim
>>>
>>> On Mon, Dec 14, 2009 at 9:53 AM, Joel Goodman <
>>> [hidden email]> wrote:
>>>
>>>>
>>>>
>>>> I think he was restarting postgres and dotcms in reverse order and not
>>>> getting the results he expected. I've sent this discussion on to him and
>>>> we're going to do some digging before we all leave for Christmas.
>>>>
>>>> Thanks for everyone's help and experiences!
>>>>
>>>> Joel
>>>>
>>>> --
>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>> www.greenville.edu
>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>
>>>> On Dec 14, 2009, at 10:38 AM, Jim Serio wrote:
>>>>
>>>>
>>>>
>>>> Joel,
>>>>
>>>> You admin should not have to restart the server - unless he doesn't know
>>>> how to restart the Postgres daemon :-)  The next time this happens, ask him
>>>> to run 'top' and see if the 'postmaster' process is the culprit.  If it is,
>>>> then our problems are related and perhaps someone at dotCMS can  Jason or
>>>> someone can increase their efforts on this.
>>>>
>>>> Regards,
>>>>    Jim
>>>>
>>>> On Mon, Dec 14, 2009 at 7:13 AM, Joel Goodman <
>>>> [hidden email]> wrote:
>>>>
>>>>>
>>>>>
>>>>> Yeah, we're running Postgres. Our Network Admin found that he has to do
>>>>> a physical machine reboot to get this to correct itself.
>>>>>
>>>>> Has this been seen on any other db platforms? We've been contemplating
>>>>> a move to MySQL and maybe this will be the silver bullet.
>>>>>
>>>>> I think our situation is really similar, if not the same judging by
>>>>> what we've observed.
>>>>>
>>>>> Thanks for the help Jim!
>>>>>
>>>>> Joel
>>>>>
>>>>> --
>>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>> www.greenville.edu
>>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>>
>>>>> On Dec 11, 2009, at 4:37 PM, Jim Serio wrote:
>>>>>
>>>>>
>>>>>
>>>>> Joel,
>>>>>
>>>>> Are you also running Postgres?  If so, then you are experiencing the
>>>>> same problem I have been for months.  The only way to resolve it, is to shut
>>>>> down dotCMS, then restart Postgres (I stop, then start) and then start back
>>>>> up dotCMS.
>>>>>
>>>>> Jason looked over my logs and a process dump a few months back and
>>>>> deduced there's something that creating some sort of SQL loop.  I have not
>>>>> been able to reliably reproduce this but I can tell you that it has happend
>>>>> on the following two occasions (since I've logged it):
>>>>>
>>>>> 1.) Someone searches for content.  The wheels will keep spinning and
>>>>> Postgres will consume all CPU cycles bumping the system load (I've seen it
>>>>> as high as 13!).
>>>>>
>>>>> 2.) Overwriting a file in the browser seems to cause the problem as
>>>>> well. It happened with a few images.
>>>>>
>>>>> Once the problem starts, the system load will never come back down
>>>>> until you restart Postgres.
>>>>>
>>>>> Of course, if you're not on Postgres, then disregard this :-)
>>>>>
>>>>> Jim
>>>>>
>>>>> On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Hey all,
>>>>>>
>>>>>> We've been having some strange server load issues with dotCMS 1.7 -
>>>>>> wanted to put some feelers out and see if any of you have a)seen this and b)
>>>>>> if an  upgrade might fix what we're seeing.
>>>>>>
>>>>>> This has only happened a few times, but basically dotCMS will use up
>>>>>> memory and keep that session open - today our load stats show 173 hours for
>>>>>> one single session. We have 3 of 8 memory bays at 100% load, another at
>>>>>> around 93%, and at that point our server stops responding and time out on
>>>>>> client browser requests.
>>>>>>
>>>>>> We're not sure what does this. We haven't changed any configuration,
>>>>>> there's plenty of storage space. Memory requests just aren't being spread
>>>>>> out across all system RAM.
>>>>>>
>>>>>>  Any help is super appreciated. It's annoying to have our site going
>>>>>> down so often after 9 months of rock-solid service.
>>>>>>
>>>>>> Joel
>>>>>>
>>>>>> --
>>>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>>> www.greenville.edu
>>>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>> --
>>
>> Met vriendelijke groet / Kind regards,
>>
>> Olaf Molenveld
>> ----------------------------------------------
>> Interactive Link
>> Keizerstraat 31
>> 3512 EA Utrecht
>> T +31 (0)30 634 32 66
>> F +31 (0)30 634 32 65
>> www.interactivelink.nl
>> [hidden email]
>>
>> INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
>> PARODOS DEEL UIT VAN DE HANSNEL GROEP
>> ----------------------------------------------
>> De informatie in dit e-mail bericht is uitsluitend bestemd voor de
>> geadresseerde.
>> Verstrekking aan en gebruik door anderen is niet toegestaan.
>> Door de elektronische verzending van het bericht kunnen er geen rechten
>> worden ontleend aan de informatie.
>>
>>
>>
>  
>



--

Met vriendelijke groet / Kind regards,

Olaf Molenveld
----------------------------------------------
Interactive Link
Keizerstraat 31
3512 EA Utrecht
T +31 (0)30 634 32 66
F +31 (0)30 634 32 65
www.interactivelink.nl
[hidden email]

INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
PARODOS DEEL UIT VAN DE HANSNEL GROEP
----------------------------------------------
De informatie in dit e-mail bericht is uitsluitend bestemd voor de
geadresseerde.
Verstrekking aan en gebruik door anderen is niet toegestaan.
Door de elektronische verzending van het bericht kunnen er geen rechten
worden ontleend aan de informatie.
Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

Jim Serio-2
Olaf,

I hope you get it resolved.  Nothing is more frustrating than not knowing
how to fix the problem.  I wonder if dotCMS is trying to do some database
work even after a restart?  One thing AI noticed was after I would restart
dotCMS/Postgres I would see in the error logs "firing off missed process" a
few dozen times.  I'm not sure if these are db-related but it seems it took
awhile for them to finish before the load went down.

Anything in the logs stand out?

Jim

On Mon, Dec 14, 2009 at 3:27 PM, Olaf iLink <[hidden email]> wrote:

>
>
> Hello Jim,
>
> nope, we didn't change anything, the problem seems to appear spontaneously.
> We already pruned the versioning, re-indexed the content, flushed all caches
> etc.etc. but still the same behaviour..
>
> Olaf
>
>
> On Mon, Dec 14, 2009 at 11:20 PM, Jim Serio <[hidden email]>wrote:
>
>>
>>
>> Olaf,
>>
>> You have Michael working on the site?  Were any changes made to it right
>> before the problem arose?  Perhaps any changes made can be reverted to see
>> if they are the cause?
>>
>> Jim
>>
>>  On Mon, Dec 14, 2009 at 3:01 PM, Olaf iLink <[hidden email]> wrote:
>>
>>>
>>>
>>> Hello,
>>>
>>> tonight we're living in a world of dotCMS pain. Our dotCMS production
>>> server decided to go bang on us, getting unresponsive and taking all the
>>> resources with it after several minutes...killing and restarting dotCMS and
>>> other daemons repeats the same problem over and over. Thanks to some deep
>>> investigations of Michiel, who's working with us on this project, we found
>>> out that the old-generation heap seams to cause our problem. It fills up
>>> real quick after startup to 100% and then the full GC kicks in. The full GC
>>> keeps on running and claims all the resources, shutting everything down. As
>>> is described here:
>>>
>>> http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp
>>>
>>> "If there is a memory leak or inadequate heap allocated, eventually the
>>> older generation will start to run out of room causing the Full GC thread to
>>> run (nearly) continuously. Since this process "stops the world", Resin won't
>>> be able to respond to requests and they'll start to back up. "
>>>
>>> Anybody any idea what to do? We're already experimenting with the
>>> heap-settings, we disabled the spellchecker, the database (mySQL) is stable
>>> and tuned allright, there is enough memory available. We're out of options
>>> right now...
>>>
>>> Olaf
>>>
>>>
>>>
>>>
>>> On Mon, Dec 14, 2009 at 6:59 PM, Joel Goodman <
>>> [hidden email]> wrote:
>>>
>>>>
>>>>
>>>> Jim, our net admin said he ran htop and said that postmaster was not the
>>>> culprit...
>>>>
>>>> Joel
>>>>
>>>> --
>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>> www.greenville.edu
>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>
>>>> On Dec 14, 2009, at 11:13 AM, Jim Serio wrote:
>>>>
>>>>
>>>>
>>>> I always worry about the order of shutting things down.  Long ago I had
>>>> to shut down dotCMS and I suspect this somehow corrupted data inside
>>>> Postgres since afterward our Tasks have never worked.  What I've done is
>>>> shut down dotCMS, then Postgres which seems logical but who knows.
>>>>
>>>> Jim
>>>>
>>>> On Mon, Dec 14, 2009 at 9:53 AM, Joel Goodman <
>>>> [hidden email]> wrote:
>>>>
>>>>>
>>>>>
>>>>> I think he was restarting postgres and dotcms in reverse order and not
>>>>> getting the results he expected. I've sent this discussion on to him and
>>>>> we're going to do some digging before we all leave for Christmas.
>>>>>
>>>>> Thanks for everyone's help and experiences!
>>>>>
>>>>> Joel
>>>>>
>>>>> --
>>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>> www.greenville.edu
>>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>>
>>>>> On Dec 14, 2009, at 10:38 AM, Jim Serio wrote:
>>>>>
>>>>>
>>>>>
>>>>> Joel,
>>>>>
>>>>> You admin should not have to restart the server - unless he doesn't
>>>>> know how to restart the Postgres daemon :-)  The next time this happens, ask
>>>>> him to run 'top' and see if the 'postmaster' process is the culprit.  If it
>>>>> is, then our problems are related and perhaps someone at dotCMS can  Jason
>>>>> or someone can increase their efforts on this.
>>>>>
>>>>> Regards,
>>>>>    Jim
>>>>>
>>>>> On Mon, Dec 14, 2009 at 7:13 AM, Joel Goodman <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Yeah, we're running Postgres. Our Network Admin found that he has to
>>>>>> do a physical machine reboot to get this to correct itself.
>>>>>>
>>>>>> Has this been seen on any other db platforms? We've been contemplating
>>>>>> a move to MySQL and maybe this will be the silver bullet.
>>>>>>
>>>>>> I think our situation is really similar, if not the same judging by
>>>>>> what we've observed.
>>>>>>
>>>>>> Thanks for the help Jim!
>>>>>>
>>>>>> Joel
>>>>>>
>>>>>> --
>>>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>>> www.greenville.edu
>>>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>>>
>>>>>> On Dec 11, 2009, at 4:37 PM, Jim Serio wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Joel,
>>>>>>
>>>>>> Are you also running Postgres?  If so, then you are experiencing the
>>>>>> same problem I have been for months.  The only way to resolve it, is to shut
>>>>>> down dotCMS, then restart Postgres (I stop, then start) and then start back
>>>>>> up dotCMS.
>>>>>>
>>>>>> Jason looked over my logs and a process dump a few months back and
>>>>>> deduced there's something that creating some sort of SQL loop.  I have not
>>>>>> been able to reliably reproduce this but I can tell you that it has happend
>>>>>> on the following two occasions (since I've logged it):
>>>>>>
>>>>>> 1.) Someone searches for content.  The wheels will keep spinning and
>>>>>> Postgres will consume all CPU cycles bumping the system load (I've seen it
>>>>>> as high as 13!).
>>>>>>
>>>>>> 2.) Overwriting a file in the browser seems to cause the problem as
>>>>>> well. It happened with a few images.
>>>>>>
>>>>>> Once the problem starts, the system load will never come back down
>>>>>> until you restart Postgres.
>>>>>>
>>>>>> Of course, if you're not on Postgres, then disregard this :-)
>>>>>>
>>>>>> Jim
>>>>>>
>>>>>> On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman <
>>>>>> [hidden email]> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hey all,
>>>>>>>
>>>>>>> We've been having some strange server load issues with dotCMS 1.7 -
>>>>>>> wanted to put some feelers out and see if any of you have a)seen this and b)
>>>>>>> if an  upgrade might fix what we're seeing.
>>>>>>>
>>>>>>> This has only happened a few times, but basically dotCMS will use up
>>>>>>> memory and keep that session open - today our load stats show 173 hours for
>>>>>>> one single session. We have 3 of 8 memory bays at 100% load, another at
>>>>>>> around 93%, and at that point our server stops responding and time out on
>>>>>>> client browser requests.
>>>>>>>
>>>>>>> We're not sure what does this. We haven't changed any configuration,
>>>>>>> there's plenty of storage space. Memory requests just aren't being spread
>>>>>>> out across all system RAM.
>>>>>>>
>>>>>>>  Any help is super appreciated. It's annoying to have our site going
>>>>>>> down so often after 9 months of rock-solid service.
>>>>>>>
>>>>>>> Joel
>>>>>>>
>>>>>>> --
>>>>>>> Joel Goodman ([hidden email]) || Web Content
>>>>>>> Coordinator
>>>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>>>> www.greenville.edu
>>>>>>> Futuristic • Strategic • Responsibility • Restorative *•*
>>>>>>>  Competition
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Met vriendelijke groet / Kind regards,
>>>
>>> Olaf Molenveld
>>> ----------------------------------------------
>>> Interactive Link
>>> Keizerstraat 31
>>> 3512 EA Utrecht
>>> T +31 (0)30 634 32 66
>>> F +31 (0)30 634 32 65
>>> www.interactivelink.nl
>>> [hidden email]
>>>
>>> INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
>>> PARODOS DEEL UIT VAN DE HANSNEL GROEP
>>> ----------------------------------------------
>>> De informatie in dit e-mail bericht is uitsluitend bestemd voor de
>>> geadresseerde.
>>> Verstrekking aan en gebruik door anderen is niet toegestaan.
>>> Door de elektronische verzending van het bericht kunnen er geen rechten
>>> worden ontleend aan de informatie.
>>>
>>>
>>>
>>
>
>
> --
>
> Met vriendelijke groet / Kind regards,
>
> Olaf Molenveld
> ----------------------------------------------
> Interactive Link
> Keizerstraat 31
> 3512 EA Utrecht
> T +31 (0)30 634 32 66
> F +31 (0)30 634 32 65
> www.interactivelink.nl
> [hidden email]
>
> INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
> PARODOS DEEL UIT VAN DE HANSNEL GROEP
> ----------------------------------------------
> De informatie in dit e-mail bericht is uitsluitend bestemd voor de
> geadresseerde.
> Verstrekking aan en gebruik door anderen is niet toegestaan.
> Door de elektronische verzending van het bericht kunnen er geen rechten
> worden ontleend aan de informatie.
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

wezell-2
Administrator
In reply to this post by olafmolilink
Off the cuff, I would say one of two things:  1) give it more memory or 2)
lower your cache region numbers.

Will

----
Will Ezell
www.dotcms.org
phone: 305.858.1422 ext. 286
direct: 978.296.4239
skype: wezell1




On Mon, Dec 14, 2009 at 5:27 PM, Olaf iLink <[hidden email]> wrote:

>
>
> Hello Jim,
>
> nope, we didn't change anything, the problem seems to appear spontaneously.
> We already pruned the versioning, re-indexed the content, flushed all caches
> etc.etc. but still the same behaviour..
>
> Olaf
>
>
> On Mon, Dec 14, 2009 at 11:20 PM, Jim Serio <[hidden email]>wrote:
>
>>
>>
>> Olaf,
>>
>> You have Michael working on the site?  Were any changes made to it right
>> before the problem arose?  Perhaps any changes made can be reverted to see
>> if they are the cause?
>>
>> Jim
>>
>>  On Mon, Dec 14, 2009 at 3:01 PM, Olaf iLink <[hidden email]> wrote:
>>
>>>
>>>
>>> Hello,
>>>
>>> tonight we're living in a world of dotCMS pain. Our dotCMS production
>>> server decided to go bang on us, getting unresponsive and taking all the
>>> resources with it after several minutes...killing and restarting dotCMS and
>>> other daemons repeats the same problem over and over. Thanks to some deep
>>> investigations of Michiel, who's working with us on this project, we found
>>> out that the old-generation heap seams to cause our problem. It fills up
>>> real quick after startup to 100% and then the full GC kicks in. The full GC
>>> keeps on running and claims all the resources, shutting everything down. As
>>> is described here:
>>>
>>> http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp
>>>
>>> "If there is a memory leak or inadequate heap allocated, eventually the
>>> older generation will start to run out of room causing the Full GC thread to
>>> run (nearly) continuously. Since this process "stops the world", Resin won't
>>> be able to respond to requests and they'll start to back up. "
>>>
>>> Anybody any idea what to do? We're already experimenting with the
>>> heap-settings, we disabled the spellchecker, the database (mySQL) is stable
>>> and tuned allright, there is enough memory available. We're out of options
>>> right now...
>>>
>>> Olaf
>>>
>>>
>>>
>>>
>>> On Mon, Dec 14, 2009 at 6:59 PM, Joel Goodman <
>>> [hidden email]> wrote:
>>>
>>>>
>>>>
>>>> Jim, our net admin said he ran htop and said that postmaster was not the
>>>> culprit...
>>>>
>>>> Joel
>>>>
>>>> --
>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>> www.greenville.edu
>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>
>>>> On Dec 14, 2009, at 11:13 AM, Jim Serio wrote:
>>>>
>>>>
>>>>
>>>> I always worry about the order of shutting things down.  Long ago I had
>>>> to shut down dotCMS and I suspect this somehow corrupted data inside
>>>> Postgres since afterward our Tasks have never worked.  What I've done is
>>>> shut down dotCMS, then Postgres which seems logical but who knows.
>>>>
>>>> Jim
>>>>
>>>> On Mon, Dec 14, 2009 at 9:53 AM, Joel Goodman <
>>>> [hidden email]> wrote:
>>>>
>>>>>
>>>>>
>>>>> I think he was restarting postgres and dotcms in reverse order and not
>>>>> getting the results he expected. I've sent this discussion on to him and
>>>>> we're going to do some digging before we all leave for Christmas.
>>>>>
>>>>> Thanks for everyone's help and experiences!
>>>>>
>>>>> Joel
>>>>>
>>>>> --
>>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>> www.greenville.edu
>>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>>
>>>>> On Dec 14, 2009, at 10:38 AM, Jim Serio wrote:
>>>>>
>>>>>
>>>>>
>>>>> Joel,
>>>>>
>>>>> You admin should not have to restart the server - unless he doesn't
>>>>> know how to restart the Postgres daemon :-)  The next time this happens, ask
>>>>> him to run 'top' and see if the 'postmaster' process is the culprit.  If it
>>>>> is, then our problems are related and perhaps someone at dotCMS can  Jason
>>>>> or someone can increase their efforts on this.
>>>>>
>>>>> Regards,
>>>>>    Jim
>>>>>
>>>>> On Mon, Dec 14, 2009 at 7:13 AM, Joel Goodman <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Yeah, we're running Postgres. Our Network Admin found that he has to
>>>>>> do a physical machine reboot to get this to correct itself.
>>>>>>
>>>>>> Has this been seen on any other db platforms? We've been contemplating
>>>>>> a move to MySQL and maybe this will be the silver bullet.
>>>>>>
>>>>>> I think our situation is really similar, if not the same judging by
>>>>>> what we've observed.
>>>>>>
>>>>>> Thanks for the help Jim!
>>>>>>
>>>>>> Joel
>>>>>>
>>>>>> --
>>>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>>> www.greenville.edu
>>>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>>>
>>>>>> On Dec 11, 2009, at 4:37 PM, Jim Serio wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Joel,
>>>>>>
>>>>>> Are you also running Postgres?  If so, then you are experiencing the
>>>>>> same problem I have been for months.  The only way to resolve it, is to shut
>>>>>> down dotCMS, then restart Postgres (I stop, then start) and then start back
>>>>>> up dotCMS.
>>>>>>
>>>>>> Jason looked over my logs and a process dump a few months back and
>>>>>> deduced there's something that creating some sort of SQL loop.  I have not
>>>>>> been able to reliably reproduce this but I can tell you that it has happend
>>>>>> on the following two occasions (since I've logged it):
>>>>>>
>>>>>> 1.) Someone searches for content.  The wheels will keep spinning and
>>>>>> Postgres will consume all CPU cycles bumping the system load (I've seen it
>>>>>> as high as 13!).
>>>>>>
>>>>>> 2.) Overwriting a file in the browser seems to cause the problem as
>>>>>> well. It happened with a few images.
>>>>>>
>>>>>> Once the problem starts, the system load will never come back down
>>>>>> until you restart Postgres.
>>>>>>
>>>>>> Of course, if you're not on Postgres, then disregard this :-)
>>>>>>
>>>>>> Jim
>>>>>>
>>>>>> On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman <
>>>>>> [hidden email]> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hey all,
>>>>>>>
>>>>>>> We've been having some strange server load issues with dotCMS 1.7 -
>>>>>>> wanted to put some feelers out and see if any of you have a)seen this and b)
>>>>>>> if an  upgrade might fix what we're seeing.
>>>>>>>
>>>>>>> This has only happened a few times, but basically dotCMS will use up
>>>>>>> memory and keep that session open - today our load stats show 173 hours for
>>>>>>> one single session. We have 3 of 8 memory bays at 100% load, another at
>>>>>>> around 93%, and at that point our server stops responding and time out on
>>>>>>> client browser requests.
>>>>>>>
>>>>>>> We're not sure what does this. We haven't changed any configuration,
>>>>>>> there's plenty of storage space. Memory requests just aren't being spread
>>>>>>> out across all system RAM.
>>>>>>>
>>>>>>>  Any help is super appreciated. It's annoying to have our site going
>>>>>>> down so often after 9 months of rock-solid service.
>>>>>>>
>>>>>>> Joel
>>>>>>>
>>>>>>> --
>>>>>>> Joel Goodman ([hidden email]) || Web Content
>>>>>>> Coordinator
>>>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>>>> www.greenville.edu
>>>>>>> Futuristic • Strategic • Responsibility • Restorative *•*
>>>>>>>  Competition
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Met vriendelijke groet / Kind regards,
>>>
>>> Olaf Molenveld
>>> ----------------------------------------------
>>> Interactive Link
>>> Keizerstraat 31
>>> 3512 EA Utrecht
>>> T +31 (0)30 634 32 66
>>> F +31 (0)30 634 32 65
>>> www.interactivelink.nl
>>> [hidden email]
>>>
>>> INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
>>> PARODOS DEEL UIT VAN DE HANSNEL GROEP
>>> ----------------------------------------------
>>> De informatie in dit e-mail bericht is uitsluitend bestemd voor de
>>> geadresseerde.
>>> Verstrekking aan en gebruik door anderen is niet toegestaan.
>>> Door de elektronische verzending van het bericht kunnen er geen rechten
>>> worden ontleend aan de informatie.
>>>
>>>
>>>
>>
>
>
> --
>
> Met vriendelijke groet / Kind regards,
>
> Olaf Molenveld
> ----------------------------------------------
> Interactive Link
> Keizerstraat 31
> 3512 EA Utrecht
> T +31 (0)30 634 32 66
> F +31 (0)30 634 32 65
> www.interactivelink.nl
> [hidden email]
>
> INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
> PARODOS DEEL UIT VAN DE HANSNEL GROEP
> ----------------------------------------------
> De informatie in dit e-mail bericht is uitsluitend bestemd voor de
> geadresseerde.
> Verstrekking aan en gebruik door anderen is niet toegestaan.
> Door de elektronische verzending van het bericht kunnen er geen rechten
> worden ontleend aan de informatie.
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

olafmolilink
In reply to this post by Jim Serio-2
we have currenty disabled all the plugins, and this seems to help. We're now
systematically trying to see what plugin might cause the problems. Good to
see we can get things more stable now.

On Mon, Dec 14, 2009 at 11:30 PM, Jim Serio <[hidden email]> wrote:

>
>
> Olaf,
>
> I hope you get it resolved.  Nothing is more frustrating than not knowing
> how to fix the problem.  I wonder if dotCMS is trying to do some database
> work even after a restart?  One thing AI noticed was after I would restart
> dotCMS/Postgres I would see in the error logs "firing off missed process" a
> few dozen times.  I'm not sure if these are db-related but it seems it took
> awhile for them to finish before the load went down.
>
> Anything in the logs stand out?
>
> Jim
>
>
> On Mon, Dec 14, 2009 at 3:27 PM, Olaf iLink <[hidden email]> wrote:
>
>>
>>
>> Hello Jim,
>>
>> nope, we didn't change anything, the problem seems to appear
>> spontaneously. We already pruned the versioning, re-indexed the content,
>> flushed all caches etc.etc. but still the same behaviour..
>>
>> Olaf
>>
>>
>> On Mon, Dec 14, 2009 at 11:20 PM, Jim Serio <[hidden email]>wrote:
>>
>>>
>>>
>>> Olaf,
>>>
>>> You have Michael working on the site?  Were any changes made to it right
>>> before the problem arose?  Perhaps any changes made can be reverted to see
>>> if they are the cause?
>>>
>>> Jim
>>>
>>>  On Mon, Dec 14, 2009 at 3:01 PM, Olaf iLink <[hidden email]> wrote:
>>>
>>>>
>>>>
>>>> Hello,
>>>>
>>>> tonight we're living in a world of dotCMS pain. Our dotCMS production
>>>> server decided to go bang on us, getting unresponsive and taking all the
>>>> resources with it after several minutes...killing and restarting dotCMS and
>>>> other daemons repeats the same problem over and over. Thanks to some deep
>>>> investigations of Michiel, who's working with us on this project, we found
>>>> out that the old-generation heap seams to cause our problem. It fills up
>>>> real quick after startup to 100% and then the full GC kicks in. The full GC
>>>> keeps on running and claims all the resources, shutting everything down. As
>>>> is described here:
>>>>
>>>> http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp
>>>>
>>>> "If there is a memory leak or inadequate heap allocated, eventually the
>>>> older generation will start to run out of room causing the Full GC thread to
>>>> run (nearly) continuously. Since this process "stops the world", Resin won't
>>>> be able to respond to requests and they'll start to back up. "
>>>>
>>>> Anybody any idea what to do? We're already experimenting with the
>>>> heap-settings, we disabled the spellchecker, the database (mySQL) is stable
>>>> and tuned allright, there is enough memory available. We're out of options
>>>> right now...
>>>>
>>>> Olaf
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Dec 14, 2009 at 6:59 PM, Joel Goodman <
>>>> [hidden email]> wrote:
>>>>
>>>>>
>>>>>
>>>>> Jim, our net admin said he ran htop and said that postmaster was not
>>>>> the culprit...
>>>>>
>>>>> Joel
>>>>>
>>>>> --
>>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>> www.greenville.edu
>>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>>
>>>>> On Dec 14, 2009, at 11:13 AM, Jim Serio wrote:
>>>>>
>>>>>
>>>>>
>>>>> I always worry about the order of shutting things down.  Long ago I had
>>>>> to shut down dotCMS and I suspect this somehow corrupted data inside
>>>>> Postgres since afterward our Tasks have never worked.  What I've done is
>>>>> shut down dotCMS, then Postgres which seems logical but who knows.
>>>>>
>>>>> Jim
>>>>>
>>>>> On Mon, Dec 14, 2009 at 9:53 AM, Joel Goodman <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> I think he was restarting postgres and dotcms in reverse order and not
>>>>>> getting the results he expected. I've sent this discussion on to him and
>>>>>> we're going to do some digging before we all leave for Christmas.
>>>>>>
>>>>>> Thanks for everyone's help and experiences!
>>>>>>
>>>>>> Joel
>>>>>>
>>>>>> --
>>>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>>> www.greenville.edu
>>>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>>>
>>>>>> On Dec 14, 2009, at 10:38 AM, Jim Serio wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Joel,
>>>>>>
>>>>>> You admin should not have to restart the server - unless he doesn't
>>>>>> know how to restart the Postgres daemon :-)  The next time this happens, ask
>>>>>> him to run 'top' and see if the 'postmaster' process is the culprit.  If it
>>>>>> is, then our problems are related and perhaps someone at dotCMS can  Jason
>>>>>> or someone can increase their efforts on this.
>>>>>>
>>>>>> Regards,
>>>>>>    Jim
>>>>>>
>>>>>> On Mon, Dec 14, 2009 at 7:13 AM, Joel Goodman <
>>>>>> [hidden email]> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Yeah, we're running Postgres. Our Network Admin found that he has to
>>>>>>> do a physical machine reboot to get this to correct itself.
>>>>>>>
>>>>>>> Has this been seen on any other db platforms? We've been
>>>>>>> contemplating a move to MySQL and maybe this will be the silver bullet.
>>>>>>>
>>>>>>> I think our situation is really similar, if not the same judging by
>>>>>>> what we've observed.
>>>>>>>
>>>>>>> Thanks for the help Jim!
>>>>>>>
>>>>>>> Joel
>>>>>>>
>>>>>>> --
>>>>>>> Joel Goodman ([hidden email]) || Web Content
>>>>>>> Coordinator
>>>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>>>> www.greenville.edu
>>>>>>> Futuristic • Strategic • Responsibility • Restorative *•*
>>>>>>>  Competition
>>>>>>>
>>>>>>> On Dec 11, 2009, at 4:37 PM, Jim Serio wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Joel,
>>>>>>>
>>>>>>> Are you also running Postgres?  If so, then you are experiencing the
>>>>>>> same problem I have been for months.  The only way to resolve it, is to shut
>>>>>>> down dotCMS, then restart Postgres (I stop, then start) and then start back
>>>>>>> up dotCMS.
>>>>>>>
>>>>>>> Jason looked over my logs and a process dump a few months back and
>>>>>>> deduced there's something that creating some sort of SQL loop.  I have not
>>>>>>> been able to reliably reproduce this but I can tell you that it has happend
>>>>>>> on the following two occasions (since I've logged it):
>>>>>>>
>>>>>>> 1.) Someone searches for content.  The wheels will keep spinning and
>>>>>>> Postgres will consume all CPU cycles bumping the system load (I've seen it
>>>>>>> as high as 13!).
>>>>>>>
>>>>>>> 2.) Overwriting a file in the browser seems to cause the problem as
>>>>>>> well. It happened with a few images.
>>>>>>>
>>>>>>> Once the problem starts, the system load will never come back down
>>>>>>> until you restart Postgres.
>>>>>>>
>>>>>>> Of course, if you're not on Postgres, then disregard this :-)
>>>>>>>
>>>>>>> Jim
>>>>>>>
>>>>>>> On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman <
>>>>>>> [hidden email]> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hey all,
>>>>>>>>
>>>>>>>> We've been having some strange server load issues with dotCMS 1.7 -
>>>>>>>> wanted to put some feelers out and see if any of you have a)seen this and b)
>>>>>>>> if an  upgrade might fix what we're seeing.
>>>>>>>>
>>>>>>>> This has only happened a few times, but basically dotCMS will use up
>>>>>>>> memory and keep that session open - today our load stats show 173 hours for
>>>>>>>> one single session. We have 3 of 8 memory bays at 100% load, another at
>>>>>>>> around 93%, and at that point our server stops responding and time out on
>>>>>>>> client browser requests.
>>>>>>>>
>>>>>>>> We're not sure what does this. We haven't changed any configuration,
>>>>>>>> there's plenty of storage space. Memory requests just aren't being spread
>>>>>>>> out across all system RAM.
>>>>>>>>
>>>>>>>>  Any help is super appreciated. It's annoying to have our site
>>>>>>>> going down so often after 9 months of rock-solid service.
>>>>>>>>
>>>>>>>> Joel
>>>>>>>>
>>>>>>>> --
>>>>>>>> Joel Goodman ([hidden email]) || Web Content
>>>>>>>> Coordinator
>>>>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>>>>> www.greenville.edu
>>>>>>>> Futuristic • Strategic • Responsibility • Restorative *•*
>>>>>>>>  Competition
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Met vriendelijke groet / Kind regards,
>>>>
>>>> Olaf Molenveld
>>>> ----------------------------------------------
>>>> Interactive Link
>>>> Keizerstraat 31
>>>> 3512 EA Utrecht
>>>> T +31 (0)30 634 32 66
>>>> F +31 (0)30 634 32 65
>>>> www.interactivelink.nl
>>>> [hidden email]
>>>>
>>>> INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
>>>> PARODOS DEEL UIT VAN DE HANSNEL GROEP
>>>> ----------------------------------------------
>>>> De informatie in dit e-mail bericht is uitsluitend bestemd voor de
>>>> geadresseerde.
>>>> Verstrekking aan en gebruik door anderen is niet toegestaan.
>>>> Door de elektronische verzending van het bericht kunnen er geen rechten
>>>> worden ontleend aan de informatie.
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Met vriendelijke groet / Kind regards,
>>
>> Olaf Molenveld
>> ----------------------------------------------
>> Interactive Link
>> Keizerstraat 31
>> 3512 EA Utrecht
>> T +31 (0)30 634 32 66
>> F +31 (0)30 634 32 65
>> www.interactivelink.nl
>> [hidden email]
>>
>> INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
>> PARODOS DEEL UIT VAN DE HANSNEL GROEP
>> ----------------------------------------------
>> De informatie in dit e-mail bericht is uitsluitend bestemd voor de
>> geadresseerde.
>> Verstrekking aan en gebruik door anderen is niet toegestaan.
>> Door de elektronische verzending van het bericht kunnen er geen rechten
>> worden ontleend aan de informatie.
>>
>>
>>
>  
>



--

Met vriendelijke groet / Kind regards,

Olaf Molenveld
----------------------------------------------
Interactive Link
Keizerstraat 31
3512 EA Utrecht
T +31 (0)30 634 32 66
F +31 (0)30 634 32 65
www.interactivelink.nl
[hidden email]

INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
PARODOS DEEL UIT VAN DE HANSNEL GROEP
----------------------------------------------
De informatie in dit e-mail bericht is uitsluitend bestemd voor de
geadresseerde.
Verstrekking aan en gebruik door anderen is niet toegestaan.
Door de elektronische verzending van het bericht kunnen er geen rechten
worden ontleend aan de informatie.
Reply | Threaded
Open this post in threaded view
|

Re: dotCMS Server Load Issues

olafmolilink
we now have re-installed all the plugins, everything is the same, but the
dotCMS seems to run smooth and stable again. Very strange, we'll keep an eye
on it..

Olaf

On Mon, Dec 14, 2009 at 11:48 PM, Olaf iLink <[hidden email]> wrote:

> we have currenty disabled all the plugins, and this seems to help. We're
> now systematically trying to see what plugin might cause the problems. Good
> to see we can get things more stable now.
>
>
> On Mon, Dec 14, 2009 at 11:30 PM, Jim Serio <[hidden email]>wrote:
>
>>
>>
>> Olaf,
>>
>> I hope you get it resolved.  Nothing is more frustrating than not knowing
>> how to fix the problem.  I wonder if dotCMS is trying to do some database
>> work even after a restart?  One thing AI noticed was after I would restart
>> dotCMS/Postgres I would see in the error logs "firing off missed process" a
>> few dozen times.  I'm not sure if these are db-related but it seems it took
>> awhile for them to finish before the load went down.
>>
>> Anything in the logs stand out?
>>
>> Jim
>>
>>
>> On Mon, Dec 14, 2009 at 3:27 PM, Olaf iLink <[hidden email]> wrote:
>>
>>>
>>>
>>> Hello Jim,
>>>
>>> nope, we didn't change anything, the problem seems to appear
>>> spontaneously. We already pruned the versioning, re-indexed the content,
>>> flushed all caches etc.etc. but still the same behaviour..
>>>
>>> Olaf
>>>
>>>
>>> On Mon, Dec 14, 2009 at 11:20 PM, Jim Serio <[hidden email]>wrote:
>>>
>>>>
>>>>
>>>> Olaf,
>>>>
>>>> You have Michael working on the site?  Were any changes made to it right
>>>> before the problem arose?  Perhaps any changes made can be reverted to see
>>>> if they are the cause?
>>>>
>>>> Jim
>>>>
>>>>  On Mon, Dec 14, 2009 at 3:01 PM, Olaf iLink <[hidden email]> wrote:
>>>>
>>>>>
>>>>>
>>>>> Hello,
>>>>>
>>>>> tonight we're living in a world of dotCMS pain. Our dotCMS production
>>>>> server decided to go bang on us, getting unresponsive and taking all the
>>>>> resources with it after several minutes...killing and restarting dotCMS and
>>>>> other daemons repeats the same problem over and over. Thanks to some deep
>>>>> investigations of Michiel, who's working with us on this project, we found
>>>>> out that the old-generation heap seams to cause our problem. It fills up
>>>>> real quick after startup to 100% and then the full GC kicks in. The full GC
>>>>> keeps on running and claims all the resources, shutting everything down. As
>>>>> is described here:
>>>>>
>>>>> http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp
>>>>>
>>>>> "If there is a memory leak or inadequate heap allocated, eventually the
>>>>> older generation will start to run out of room causing the Full GC thread to
>>>>> run (nearly) continuously. Since this process "stops the world", Resin won't
>>>>> be able to respond to requests and they'll start to back up. "
>>>>>
>>>>> Anybody any idea what to do? We're already experimenting with the
>>>>> heap-settings, we disabled the spellchecker, the database (mySQL) is stable
>>>>> and tuned allright, there is enough memory available. We're out of options
>>>>> right now...
>>>>>
>>>>> Olaf
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Dec 14, 2009 at 6:59 PM, Joel Goodman <
>>>>> [hidden email]> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Jim, our net admin said he ran htop and said that postmaster was not
>>>>>> the culprit...
>>>>>>
>>>>>> Joel
>>>>>>
>>>>>> --
>>>>>> Joel Goodman ([hidden email]) || Web Content Coordinator
>>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>>> www.greenville.edu
>>>>>> Futuristic • Strategic • Responsibility • Restorative *•* Competition
>>>>>>
>>>>>> On Dec 14, 2009, at 11:13 AM, Jim Serio wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> I always worry about the order of shutting things down.  Long ago I
>>>>>> had to shut down dotCMS and I suspect this somehow corrupted data inside
>>>>>> Postgres since afterward our Tasks have never worked.  What I've done is
>>>>>> shut down dotCMS, then Postgres which seems logical but who knows.
>>>>>>
>>>>>> Jim
>>>>>>
>>>>>> On Mon, Dec 14, 2009 at 9:53 AM, Joel Goodman <
>>>>>> [hidden email]> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I think he was restarting postgres and dotcms in reverse order and
>>>>>>> not getting the results he expected. I've sent this discussion on to him and
>>>>>>> we're going to do some digging before we all leave for Christmas.
>>>>>>>
>>>>>>> Thanks for everyone's help and experiences!
>>>>>>>
>>>>>>> Joel
>>>>>>>
>>>>>>> --
>>>>>>> Joel Goodman ([hidden email]) || Web Content
>>>>>>> Coordinator
>>>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>>>> www.greenville.edu
>>>>>>> Futuristic • Strategic • Responsibility • Restorative *•*
>>>>>>>  Competition
>>>>>>>
>>>>>>> On Dec 14, 2009, at 10:38 AM, Jim Serio wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Joel,
>>>>>>>
>>>>>>> You admin should not have to restart the server - unless he doesn't
>>>>>>> know how to restart the Postgres daemon :-)  The next time this happens, ask
>>>>>>> him to run 'top' and see if the 'postmaster' process is the culprit.  If it
>>>>>>> is, then our problems are related and perhaps someone at dotCMS can  Jason
>>>>>>> or someone can increase their efforts on this.
>>>>>>>
>>>>>>> Regards,
>>>>>>>    Jim
>>>>>>>
>>>>>>> On Mon, Dec 14, 2009 at 7:13 AM, Joel Goodman <
>>>>>>> [hidden email]> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Yeah, we're running Postgres. Our Network Admin found that he has to
>>>>>>>> do a physical machine reboot to get this to correct itself.
>>>>>>>>
>>>>>>>> Has this been seen on any other db platforms? We've been
>>>>>>>> contemplating a move to MySQL and maybe this will be the silver bullet.
>>>>>>>>
>>>>>>>> I think our situation is really similar, if not the same judging by
>>>>>>>> what we've observed.
>>>>>>>>
>>>>>>>> Thanks for the help Jim!
>>>>>>>>
>>>>>>>> Joel
>>>>>>>>
>>>>>>>> --
>>>>>>>> Joel Goodman ([hidden email]) || Web Content
>>>>>>>> Coordinator
>>>>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>>>>> www.greenville.edu
>>>>>>>> Futuristic • Strategic • Responsibility • Restorative *•*
>>>>>>>>  Competition
>>>>>>>>
>>>>>>>> On Dec 11, 2009, at 4:37 PM, Jim Serio wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Joel,
>>>>>>>>
>>>>>>>> Are you also running Postgres?  If so, then you are experiencing the
>>>>>>>> same problem I have been for months.  The only way to resolve it, is to shut
>>>>>>>> down dotCMS, then restart Postgres (I stop, then start) and then start back
>>>>>>>> up dotCMS.
>>>>>>>>
>>>>>>>> Jason looked over my logs and a process dump a few months back and
>>>>>>>> deduced there's something that creating some sort of SQL loop.  I have not
>>>>>>>> been able to reliably reproduce this but I can tell you that it has happend
>>>>>>>> on the following two occasions (since I've logged it):
>>>>>>>>
>>>>>>>> 1.) Someone searches for content.  The wheels will keep spinning and
>>>>>>>> Postgres will consume all CPU cycles bumping the system load (I've seen it
>>>>>>>> as high as 13!).
>>>>>>>>
>>>>>>>> 2.) Overwriting a file in the browser seems to cause the problem as
>>>>>>>> well. It happened with a few images.
>>>>>>>>
>>>>>>>> Once the problem starts, the system load will never come back down
>>>>>>>> until you restart Postgres.
>>>>>>>>
>>>>>>>> Of course, if you're not on Postgres, then disregard this :-)
>>>>>>>>
>>>>>>>> Jim
>>>>>>>>
>>>>>>>> On Fri, Dec 11, 2009 at 1:41 PM, Joel Goodman <
>>>>>>>> [hidden email]> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hey all,
>>>>>>>>>
>>>>>>>>> We've been having some strange server load issues with dotCMS 1.7 -
>>>>>>>>> wanted to put some feelers out and see if any of you have a)seen this and b)
>>>>>>>>> if an  upgrade might fix what we're seeing.
>>>>>>>>>
>>>>>>>>> This has only happened a few times, but basically dotCMS will use
>>>>>>>>> up memory and keep that session open - today our load stats show 173 hours
>>>>>>>>> for one single session. We have 3 of 8 memory bays at 100% load, another at
>>>>>>>>> around 93%, and at that point our server stops responding and time out on
>>>>>>>>> client browser requests.
>>>>>>>>>
>>>>>>>>> We're not sure what does this. We haven't changed any
>>>>>>>>> configuration, there's plenty of storage space. Memory requests just aren't
>>>>>>>>> being spread out across all system RAM.
>>>>>>>>>
>>>>>>>>>  Any help is super appreciated. It's annoying to have our site
>>>>>>>>> going down so often after 9 months of rock-solid service.
>>>>>>>>>
>>>>>>>>> Joel
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Joel Goodman ([hidden email]) || Web Content
>>>>>>>>> Coordinator
>>>>>>>>> 315 E College Ave | Greenville, IL 62246 | (618) 664-6515 |
>>>>>>>>> www.greenville.edu
>>>>>>>>> Futuristic • Strategic • Responsibility • Restorative *•*
>>>>>>>>>  Competition
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Met vriendelijke groet / Kind regards,
>>>>>
>>>>> Olaf Molenveld
>>>>> ----------------------------------------------
>>>>> Interactive Link
>>>>> Keizerstraat 31
>>>>> 3512 EA Utrecht
>>>>> T +31 (0)30 634 32 66
>>>>> F +31 (0)30 634 32 65
>>>>> www.interactivelink.nl
>>>>> [hidden email]
>>>>>
>>>>> INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
>>>>> PARODOS DEEL UIT VAN DE HANSNEL GROEP
>>>>> ----------------------------------------------
>>>>> De informatie in dit e-mail bericht is uitsluitend bestemd voor de
>>>>> geadresseerde.
>>>>> Verstrekking aan en gebruik door anderen is niet toegestaan.
>>>>> Door de elektronische verzending van het bericht kunnen er geen rechten
>>>>> worden ontleend aan de informatie.
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Met vriendelijke groet / Kind regards,
>>>
>>> Olaf Molenveld
>>> ----------------------------------------------
>>> Interactive Link
>>> Keizerstraat 31
>>> 3512 EA Utrecht
>>> T +31 (0)30 634 32 66
>>> F +31 (0)30 634 32 65
>>> www.interactivelink.nl
>>> [hidden email]
>>>
>>> INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
>>> PARODOS DEEL UIT VAN DE HANSNEL GROEP
>>> ----------------------------------------------
>>> De informatie in dit e-mail bericht is uitsluitend bestemd voor de
>>> geadresseerde.
>>> Verstrekking aan en gebruik door anderen is niet toegestaan.
>>> Door de elektronische verzending van het bericht kunnen er geen rechten
>>> worden ontleend aan de informatie.
>>>
>>>
>>>
>>  
>>
>
>
>
> --
>
> Met vriendelijke groet / Kind regards,
>
> Olaf Molenveld
> ----------------------------------------------
> Interactive Link
> Keizerstraat 31
> 3512 EA Utrecht
> T +31 (0)30 634 32 66
> F +31 (0)30 634 32 65
> www.interactivelink.nl
> [hidden email]
>
> INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
> PARODOS DEEL UIT VAN DE HANSNEL GROEP
> ----------------------------------------------
> De informatie in dit e-mail bericht is uitsluitend bestemd voor de
> geadresseerde.
> Verstrekking aan en gebruik door anderen is niet toegestaan.
> Door de elektronische verzending van het bericht kunnen er geen rechten
> worden ontleend aan de informatie.
>



--

Met vriendelijke groet / Kind regards,

Olaf Molenveld
----------------------------------------------
Interactive Link
Keizerstraat 31
3512 EA Utrecht
T +31 (0)30 634 32 66
F +31 (0)30 634 32 65
www.interactivelink.nl
[hidden email]

INTERACTIVE LINK MAAKT SAMEN MET HANSNEL COMMUNICATIE, CHANNEL PANEL EN
PARODOS DEEL UIT VAN DE HANSNEL GROEP
----------------------------------------------
De informatie in dit e-mail bericht is uitsluitend bestemd voor de
geadresseerde.
Verstrekking aan en gebruik door anderen is niet toegestaan.
Door de elektronische verzending van het bericht kunnen er geen rechten
worden ontleend aan de informatie.
123