[spip-dev] SPIP, flock(), FastCGI and PHP limits

Thomas_Sutton · Août 25, 2009, 6:25

Hi all,

I am sorry for posting in English, but all of my French speakers are away on holidays at the moment.

I have encountered a problem on several of our SPIP web-sites. It manifests in several ways:

1. A request will "block" when trying to regenerate a page. The request never seems to finish.
2. SPIP will cause a PHP interpreter to hang forever. This can and has caused the web server cluster to fail to start properly (FastCGI process manager will not start new PHP processes because the old processes haven't died, but they aren't accepting requests either)!

These both seem to be problems with locking related to the SPIP cache.

1. SPIP uses flock() on a file and blocks waiting for exclusive access.
2. PHP max_execution_time is encountered and PHP ends the script.
3. The lock is still held and the interpreter never dies.

This appears to be related to this PHP bug <http://bugs.php.net/bug.php?id=47640 >, but the PHP developers don't seem interested in fixing it.

The "block" problem happened a lot with one of our SPIP 2.0.3 sites but seems to have been resolved in 2.0.8.

And the "hung interpreters" problem happen so much with a 1.9.2d that it took down our hosting provider's web-server cluster. Upgrading it to a 2.0 version seemed to resolve the problem or, we haven't noticed it again.

Are these problems known and fixed? Or is it coincidence that recent upgrades have "fixed" them?

Thank you,

Thomas Sutton
Web Developer

bouncingorange
graphic+web design

cerdic · Août 25, 2009, 12:17

Hi,

in case the trouble again occurs, you may try the alternative lock function (originaly designed for NFS) that is not based on flock function. It's in file ecrire/inc/nfslock.php

You can activate it with following define in your mes_options.php file :

define('_SPIP_LOCK_MODE',2); // utiliser le nfslock de spip

It has not been extensively tested, but i used it on SPIP-Contrib for a while with no problem, so it should work, and maybe it will allow to avoid the bug your describing.

I never had trouble with such a bug.

Cédric

Thomas_Sutton · Août 26, 2009, 1:13

Hi Cédric,

Thank you for your reply.

Hi,

in case the trouble again occurs, you may try the alternative lock function (originaly designed for NFS) that is not based on flock function. It's in file ecrire/inc/nfslock.php

You can activate it with following define in your mes_options.php file :

define('_SPIP_LOCK_MODE',2); // utiliser le nfslock de spip

It has not been extensively tested, but i used it on SPIP-Contrib for a while with no problem, so it should work, and maybe it will allow to avoid the bug your describing.

I never had trouble with such a bug.

I will try this solution if we see the problem again (most of our sites are hosted on a server cluster which uses NFS storage!)

We've only seen these problems on two sites which hit max_execution_time very frequently (usually because they have lots of large images and use lots of image_* filters), so it isn't a *huge* problem for us but it still annoys the clients.

Is there a list of these constants somewhere? I've seen all sorts of interesting things that we can do by define()ing a constant, but no list of them.

Thank you,

Thomas Sutton
Web Developer

bouncingorange
graphic+web design

azerttyu · Août 26, 2009, 7:23

Hi

Is there a list of these constants somewhere? I've seen all sorts of
interesting things that we can do by define()ing a constant, but no list of
them.

Actually, it don't exist any page about define()ing constant. You can
open a new page on doc.spip.org. It's (amha) the better way to open
this work.

Km

Thomas_Sutton · Octobre 6, 2009, 3:26

Hi Cédric,

I've just encountered this locking problem again on one of our sites. It resulted in the all requests for the backend and frontend pages being deadlocked on multiple servers. I had the server admin check and multiple PHP5 FastCGI processes were stopped in a flock() on tmp/cache/sql_desc.txt and when he killed them all the site seemed to recover.

I added the define() you suggested to the site's config/mes_options.php, but it raised errors in spip_nfslock() when trying to link() the new file as the lock:

Warning: link() [function.link]: File exists in /home/giardini.com.au/htdocs/private/inc/nfslock.php on line 126

... repeated 9 more times...

Do I need to delete the cache, etc. when I add this define()? Or should it just work?

Cheers,

Thomas Sutton
Web Developer

bouncingorange
graphic+web design

cerdic · Octobre 6, 2009, 8:21

Hi Thomas,

Hi Cédric,

I've just encountered this locking problem again on one of our sites. It resulted in the all requests for the backend and frontend pages being deadlocked on multiple servers. I had the server admin check and multiple PHP5 FastCGI processes were stopped in a flock() on tmp/cache/sql_desc.txt and when he killed them all the site seemed to recover.

I added the define() you suggested to the site's config/mes_options.php, but it raised errors in spip_nfslock() when trying to link() the new file as the lock:

Warning: link() [function.link]: File exists in /home/giardini.com.au/htdocs/private/inc/nfslock.php on line 126

ok, here we should add a @ to link function in order to avoid warning when trying to get lock
- if (link($tpath, $lock_file) == 1) {
+ if (@link($tpath, $lock_file) == 1) {

... repeated 9 more times...

not good : it means that all of the 10 tries failed and the process failed to gain lock.

Do I need to delete the cache, etc. when I add this define()? Or should it just work?

It should works, except the missing @.
But the repeated error when trying to get lock looks like there is the same trouble with this alternative locking function than with native flock function.

Cédric

Thomas_Sutton · Octobre 7, 2009, 2:47

Hi Cédric,

Thanks for your assistance.

Hi Thomas,

Hi Cédric,

I've just encountered this locking problem again on one of our sites. It resulted in the all requests for the backend and frontend pages being deadlocked on multiple servers. I had the server admin check and multiple PHP5 FastCGI processes were stopped in a flock() on tmp/cache/sql_desc.txt and when he killed them all the site seemed to recover.

I added the define() you suggested to the site's config/mes_options.php, but it raised errors in spip_nfslock() when trying to link() the new file as the lock:

Warning: link() [function.link]: File exists in /home/giardini.com.au/htdocs/private/inc/nfslock.php on line 126

ok, here we should add a @ to link function in order to avoid warning when trying to get lock
- if (link($tpath, $lock_file) == 1) {
+ if (@link($tpath, $lock_file) == 1) {

This suppresses the error message, but if it the system is deadlocked, then it's just going to fail ten times and never manage to acquire a lock. What will SPIP do when this happens? Stop with an error? Continue without holding the lock?

The problem is not that flock() does not work; it does and processes block waiting for access to the file. The problem is that SPIP seems to be locking too much for too long and deadlocking (or perhaps just an extreme case of resource starvation).

I still don't understand why all this locking is going on, especially when it's just caching data. If all this lock is doing is serialising access to update the cached file, why not use an atomic operation instead? rename(), for example, is atomic according to POSIX (and ISO C, I believe) and even over NFS. If all the process is doing is updating a cache file, surely it can write to a unique file, then rename() it into place?

... repeated 9 more times...

not good : it means that all of the 10 tries failed and the process failed to gain lock.

Do I need to delete the cache, etc. when I add this define()? Or should it just work?

It should works, except the missing @.
But the repeated error when trying to get lock looks like there is the same trouble with this alternative locking function than with native flock function.

Yeah, it looks like a deadlock to me (unless generating this data and then serialising it to a file takes many many minutes). Does SPIP ever attempt to acquire two locks at a time? If so, what is the protocol for acquiring and releasing them? If it happens again, I'll try to convince the server admin to strace all the PHP processes and figure out what locks they're holding.

Cheers,

Thomas Sutton
Web Developer

bouncingorange
graphic+web design