armtuk: Cheetah (Default)
So stracing postgres shows that my function deg2rad was segfaulting. Wow - apparently some query is calling deg2rad and passing in a null value. The function doesn't like that, so I've put in a test to make sure if it gets a null, it returns a null. So far it's looking promising. I could now submit the admin page that I was using before to test the problem, so my reproducable case has gone away. The other application that was doing it lots was trac. I'm not sure how trac's crashes can be attributed to my stored procedure in an unrelated database, but a bit of clicking around in trac has shown no problems yet. Tomorrow will be the big test. It seems to happen during the day when the server is under load much more prevalently.
armtuk: Cheetah (Default)
We are having a major malfunction with our database server that started about three days ago. For some reason database connections are closing unexpectedly or at least reaching EOF when they shouldn't be.

I did all the normal stuff. Downgraded Postgresql from 8.3 back to 8.2 wondering if that was it. That didn't fix the problem. I rebooted the server, that didn't help. Restarted everything like ten times. I posted to the PgSQL mailing list, and I posted a ticket with Rackspace. The PgSQL mailing was as always fairly helpful, but Rackspace was worthless. If there was anybody else who even answered customer support requests on a reliable basis and gave bigger servers, I would totally ditch Rackspace, but they are the best of everyone we've worked with which is sad. Not much of a recipient of fanatical support so far (ServerBeach has been excellent, but they don't really provide the kind of server we need).

After few responses that were helpful in fixing the problem I decided that the best next step was to strace tomcat to figure out what was happening on the socket. The first strace the socket threw a SIGPIPE and tomcat's thread Segfaulted :( (but didn't throw a Java exception so it looked like nothing happened from the log file). Unfortunately I wasn't expecting tomcat to crash (we've seen the DB process segfault before, but not tomcat) so I wasn't running with a non zero core dump limit.

The next runs revealed little of anything. Then I got one that had more clear information in it. It looks like tomcat is sending a database request to postgres and the recv comes back with a 0 return value, which means postgresql is closing the connection. Super wierd.

I've posted this info to the Postgres mailing list. Not sure what else I can do at this point. Maybe my postgresql.conf is stupid, but I haven't changed it recently I don't think, so I don't know why this started happening all of a sudden.

I think my next step will be to strace postgresql, but that means setting up a new instance because I can't really strace the production instance of postgresql as that would generate so much log, it would be impossible to go through it all.

Postgresql

Dec. 21st, 2007 09:56 pm
armtuk: Cheetah (Default)
Apparently Postgresql doesn't compile PLPGSQL functions:

code for the curious )

The C version of the same function is nearly 10 times faster than the plpgsql. Disturbing.

Note to self: write functions for Postgresql in C not PLPGSQL.

Profile

armtuk: Cheetah (Default)
armtuk

April 2017

S M T W T F S
      1
2345678
9101112131415
16171819 202122
23242526272829
30      

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Jul. 20th, 2017 12:51 pm
Powered by Dreamwidth Studios