Page Index Toggle Pages: 1 [2] 3  Send Topic Send Topic Print Print
Very Hot Topic (More than 25 Replies) [RESOLVED] Server has detected a low memory issue (Read 7439 times)
Steve_in_Texas
Senior Member
*****
Offline


No personal text

Posts: 893
Location: San Antonio
Joined: Feb 21st, 2004
Re: Server has detected a low memory issue...CRASH
Reply #15 - Nov 16th, 2010 at 5:40pm
Print Post Print Post  
Mark, I failed to mention that Sesame server does use about 80MB of memory when only the Inven-test.db ap is loaded. But we also load 3 other (larger) aps as well, which pushes the memory usage of sesame server upwards of around 500MB.

However, the second issue may be worth investigating; on our server pc, the sesame server GUI seems to always shows 'Physical Memory Remaining' at a constant 4,294,967,295 which is exactly the same number as the 'virtual memory remaining' shown just below it.

on my Windows 7 PC (xeon 2ghz 64-bit with 4GB memory) with the same aps loaded in client/server configuration:

Sesame server GUI shows an ever changing "Physical Memory Remaining' number. (currently at 1,838,039,040) and Virtual Mem. Remain' at 4,294,967,295.

Of note, on the server computer:
In Windows setting, 'processor scheduling' is set to 'background services', instead of 'programs'.
Also, 'Memory Usage' is set to 'system cache' instead of 'programs'.
'Virtual memory' page file is set to 'system managed' and is recommended 12,285MB but is currently set to 8190MB.

on my pc:
Processor scheduling is set to 'programs'
virtual memory recommended is 6139MB, but currently set to 'system managed' at 4,093MB.


Do you think our server has a memory issue and is ALWAYS swapping?




  
Back to top
IP Logged
 
The Cow
YaBB Administrator
*****
Offline



Posts: 2530
Joined: Nov 22nd, 2002
Re: Server has detected a low memory issue...CRASH
Reply #16 - Nov 16th, 2010 at 6:13pm
Print Post Print Post  
Steve, I think we may be seeing a fault at the point where the 32 bit application and 64 bit operating system meet. Sesame is checking for memory based on a 32 bit value. The operating system reports the amount, but - because it is large, it exceeds the bit space of a 32 bit integer and "rolls over" to a negative number. My only hesitancy here is that I have a 64 bit computers as well, running Vista and Ubuntu Linux for 64 bits and have never seen this.

I will double check the code for 32 bit variables trying to hold 64 bit values. In the meantime, I would suggest that you set some of your server's settings closer to those on a box that is not having this problem and see if that has any effect.
  

Mark Lasersohn&&Programmer&&Lantica Software, LLC
Back to top
IP Logged
 
Steve_in_Texas
Senior Member
*****
Offline


No personal text

Posts: 893
Location: San Antonio
Joined: Feb 21st, 2004
Re: Server has detected a low memory issue...CRASH
Reply #17 - Nov 16th, 2010 at 6:35pm
Print Post Print Post  
On my windows 7 pc, I was able to export ALL records without crashing! That is great to see.

I'll change the memory and cpu settings on the server machine to mimic my windows 7 machine and report back soon.

Thanks again!
  
Back to top
IP Logged
 
Steve_in_Texas
Senior Member
*****
Offline


No personal text

Posts: 893
Location: San Antonio
Joined: Feb 21st, 2004
Re: Server has detected a low memory issue...CRASH
Reply #18 - Nov 16th, 2010 at 7:48pm
Print Post Print Post  
We crashed again while trying to load Sdesigner in addition to our other aps. Since everyone was disconnected, I took the opportunity to change the server pc settings to match my pc  (system memory and processor now set to 'programs') and rebooted the machine.

With all aps unlocked and loaded back up, I don't see any difference, except the windows console is a bit sluggish to respond (not a problem). Also, the sesame GUI again shows 'physical memory remaining' unchanging at 4,294,967,295, matching 'Virtual memory remaining'...quite different than my windows 7 pc. (both machines are 64bit OS).

I hope we are getting close to finding the culprit.

Thanks for all your efforts.

Steve

  
Back to top
IP Logged
 
The Cow
YaBB Administrator
*****
Offline



Posts: 2530
Joined: Nov 22nd, 2002
Re: Server has detected a low memory issue...CRASH
Reply #19 - Nov 16th, 2010 at 8:42pm
Print Post Print Post  
I found a culprit, though I am not sure if it is "the" culprit. MS's original function to return memory information is very unhappy with machines with more than 2 GB and can report the numbers back incorrectly. They have an "extended" function that is supposed to fix this problem. They have a strange habit of adding new functions rather than fixing the old ones.

So, I changed Sesame to use the newer function. It will be in the next release. However, I can't really see how this is likely to be causing the problem. This is used to report the amount of memory available and throw a warning if the numbers get very low, which may account for your erroneous warning. But, it doesn't actually change any behavior if it does throw the warning. The only place these number get used, as opposed to simply shown, is when you are loading an application file. It determines whether you need to use an incremental load routine.

But it may be that you are actually running out of memory that a 32 bit program can address and, because the original function can't deal with more than 2 GB, it isn't reporting correctly that memory is that low. Tomorrow, I will look into the export routine for any memory hogs. You should look through the other programs running on that server (using task manager) and see if there any that are extra-large.
  

Mark Lasersohn&&Programmer&&Lantica Software, LLC
Back to top
IP Logged
 
Steve_in_Texas
Senior Member
*****
Offline


No personal text

Posts: 893
Location: San Antonio
Joined: Feb 21st, 2004
Re: Server has detected a low memory issue...CRASH
Reply #20 - Nov 16th, 2010 at 9:14pm
Print Post Print Post  
Other programs running on the server pc seem to use very little memory. The second biggest memory user is SearchIndexer.exe using 125K and then w3dbsmgr.exe using 76K and then Outlook.exe using 50K

Taskmgr shows total 'available' memory at 6.8GB and 'system cache' at 1.3 GB

Kernal memory shows total at 100,300 and paged 60,260 and nonpaged 40,000.

Seems like we have plenty of memory just waiting to be used.

I've never been keen to having all of our aps loaded into one sesame server. In the case of a crash in one database, everything hits the floor. Can we launch a second sesame server on this pc to run some of our bigger aps? They will need to have xlookups run between them, which would cause issues?

Thanks.
  
Back to top
IP Logged
 
The Cow
YaBB Administrator
*****
Offline



Posts: 2530
Joined: Nov 22nd, 2002
Re: Server has detected a low memory issue...CRASH
Reply #21 - Nov 16th, 2010 at 9:41pm
Print Post Print Post  
Steve_in_Texas wrote on Nov 16th, 2010 at 9:14pm:
Can we launch a second sesame server on this pc to run some of our bigger aps? They will need to have xlookups run between them, which would cause issues?


You can run multiple Sesame servers on one computer. Which would normally allow each of the Sesame servers more memory. Simply, assign each to a different set of ports. X commands will be a problem and will likely result in a similar use of memory.

Can you give me a run down of all of the apps/databases you are running, file sizes and "in-memory" sizes?
  

Mark Lasersohn&&Programmer&&Lantica Software, LLC
Back to top
IP Logged
 
Steve_in_Texas
Senior Member
*****
Offline


No personal text

Posts: 893
Location: San Antonio
Joined: Feb 21st, 2004
Re: Server has detected a low memory issue...CRASH
Reply #22 - Nov 17th, 2010 at 12:47am
Print Post Print Post  
Mark, here is list of our aps that load into one sesame server:

I restarted the sesame server gui and loaded up each ap, making notes of 'mem usage' from task manager on the server.

Orders.db
Filesize 166MB 
Mem usage 190MB

Products.db
Filesize 144MB
Mem usage 166MB

Inven-test.db
Filesize 63MB
Mem usage 70MB

Stock.db
filesize 10MB
mem usage 12MB

OldSales.db (only used occasionally. contains 353,000 records. no X commands.)
Filesize 189MB
Mem usage 226MB
(sesame.exe mem usage swelled to 1GB during loading phase.)

With the above aps loaded, Mem usage is sitting at 664K and issues begin to snowball....

Now that these aps are loaded, I cannot get an additional client window to appear to load the zipcodes ap. Connections show 9 of 13 (but I only have 5 client GUI's 'visible' on my pc.)

After closing all sesame clients, connections shows 4 of 13. Don't know how to release the 4 that are still 'connected'. (This happens often.)

Will run same test on my windows 7 pc.





  
Back to top
IP Logged
 
Steve_in_Texas
Senior Member
*****
Offline


No personal text

Posts: 893
Location: San Antonio
Joined: Feb 21st, 2004
Re: Server has detected a low memory issue...CRASH
Reply #23 - Nov 17th, 2010 at 1:09am
Print Post Print Post  
On my Windows 7 pc. I was able to load all of the above, including an additional ap (a backup of my Products.db.) However, after loading 6 aps, I cannot get another client GUI to appear. As before, client seats are maxed out, but with only 6 'visible' connections out of 13 max)

Taskmanager shows Sesame.exe mem usage at 845K steady.

Sesame server GUI shows physical mem remaining: 1,510,281,200

Are we getting warmer? (should I run the same test on a 32-bit OS?



  
Back to top
IP Logged
 
The Cow
YaBB Administrator
*****
Offline



Posts: 2530
Joined: Nov 22nd, 2002
Re: Server has detected a low memory issue...CRASH
Reply #24 - Nov 17th, 2010 at 2:40am
Print Post Print Post  
Sorry to put you through all of this, but I am getting truly perplexed. Could you please try some experiments? First, try to connect your max number of clients without loading any apps. After they have connected, disconnect them making sure the connection count goes down with each. If that goes as it should, try the same test with any one of the more likely suspect apps loaded. If that goes well, try loading all of the apps that are regularly loaded, but do not open any forms (do you have any "application open" programming?) and see how that goes.

From the numbers you are telling me, it sounds like you are approaching but not exceeding the 1 GB mark. We have quite a few customers with much larger applications with many more connections, so I am still suspicious that something more is going on here than mere size, especially in light of the network connection problem you are outlining.

If you do have a 32 bit OS machine that is not otherwise occupied, it might be worth a few moments to see how it compares. It might be worth a moment to get Ray or Ben on your server with remote desktop so they can see this first hand. Or, if you prefer, send in all the apps (neatly packaged) and I will run them on my machine, which sounds very similar to yours, and hopefully will exhibit the same symptoms.

BTW: Does your server log indicate any problems?
  

Mark Lasersohn&&Programmer&&Lantica Software, LLC
Back to top
IP Logged
 
Steve_in_Texas
Senior Member
*****
Offline


No personal text

Posts: 893
Location: San Antonio
Joined: Feb 21st, 2004
Re: Server has detected a low memory issue...CRASH
Reply #25 - Nov 17th, 2010 at 3:27pm
Print Post Print Post  
Mark, no problem running these tests. We're at the point that we NEED to spend the time figuring out this issue. I'm grateful for your help.

I'll run these tests tonight when I have full use of the server. I'll get back to you.
  
Back to top
IP Logged
 
Steve_in_Texas
Senior Member
*****
Offline


No personal text

Posts: 893
Location: San Antonio
Joined: Feb 21st, 2004
Re: Server has detected a low memory issue...CRASH
Reply #26 - Nov 18th, 2010 at 2:44am
Print Post Print Post  
OK Mark, per your request, here are the results of my test....

In client/server mode serving from the windows server 2003 computer: (i have 13 client seats.)

With no aps loaded, all 13 client GUI's loaded and unloaded with no problems.

I loaded Orders.db and then opened 12 additional client gui's. All 12 opened and closed with no problem. (this one has 'application open programming', which also loaded a tiny db called Pers.db)

Loaded Products.db, in addition to orders.db. 11 client gui's opened and closed just fine. (this also has ap open programming)

Loaded Inven-test.db in addition to above two db's. 10 client's opened and closed fine. (has a default form, but no ap open programming)

Loaded Stock.db and 9 clients opened and closed fine.

Ran the same test for Oldsales.db, 8 clients opened and closed fine.

Loaded zipcodes.db (wow! couldn't do this last night.) but then could not open any new client GUI's but server shows I am connecting each time I launch from icon.

closed Zipcodes.db client, and it released from the sever (as per server admin panel>unload application) and I could then open up another client GUI to load zipcodes again. I'm now showing 13 connections at the server, with only 6 client GUI's on my desktop.

I closed all my GUI's and sesame shows 8 of 13 connections. I am able to open FOUR new client Gui's. Closed them all. Sesame server now shows 9 of 13 connections.

I opened 4 more client GUI's and then they all closed and sesame server closed and popped up numerous (probably 13) notices on my desktop that the client was no longer communicating with the server.

I repeated the above test, loading zipcodes.db third in the list of aps.

After loading zipcodes.db, and then Oldsales.db. Getting a new GUI to open is getting hairy now. one gui did not open, but the next one did.

After loading Orders.db, I can't get any more GUI's. Sesame memory in taskmgr on the server shows 650,000K. Sesame server shows 13 connections. I have 7 GUI's visible on my desktop.

I can run the same test on a 32bit machine tomorrow and I'm fine with any other steps you think we should take. We need to figure this out, so just let me know what's next.

Thanks again for all your time.

Steve



  
Back to top
IP Logged
 
Steve_in_Texas
Senior Member
*****
Offline


No personal text

Posts: 893
Location: San Antonio
Joined: Feb 21st, 2004
Re: Server has detected a low memory issue...CRASH
Reply #27 - Nov 18th, 2010 at 2:51am
Print Post Print Post  
ps. I dont know what to look for in the server log, but I do see this:

"Check connection call failed."

Also, upon shutting down this test, I had to use the admin panel to 'shutdown sesame server' it would not respond for several minutes then finally crashed and popped up several notices about network communication errors and such. Happened to fast to write it all down.

Steve
  
Back to top
IP Logged
 
The Cow
YaBB Administrator
*****
Offline



Posts: 2530
Joined: Nov 22nd, 2002
Re: Server has detected a low memory issue...CRASH
Reply #28 - Nov 18th, 2010 at 1:35pm
Print Post Print Post  
After some experiments here, and with the numbers you have posted, I am forming a picture. I can create similar symptoms if I take memory usage to extremes. I opened 24 different copies of your app, using 24 clients. Then, with Sesame server memory on the edge of 2 GB of usage, I kicked off enough exports to kick the usage over 2 GB. At which point, memory allocations start to fail and, because the conditions are notably controlled, the failures were noted and logged, I can see how in some cases they could result in a dropped connections and communications cessation. Basically, if there is no memory, there is no memory to use to report that there is no memory.

When you report memory usage of Sesame server from task manager, are you reporting the column labeled, "Working Set (Memory)"?
  

Mark Lasersohn&&Programmer&&Lantica Software, LLC
Back to top
IP Logged
 
Steve_in_Texas
Senior Member
*****
Offline


No personal text

Posts: 893
Location: San Antonio
Joined: Feb 21st, 2004
Re: Server has detected a low memory issue...CRASH
Reply #29 - Nov 18th, 2010 at 3:08pm
Print Post Print Post  
I sent you an email with a screenshot.

I don't think Windows server 2003 has a 'working set' column. Instead, it's labeled 'Mem Usage'.

Thanks.
  
Back to top
IP Logged
 
Page Index Toggle Pages: 1 [2] 3 
Send Topic Send Topic Print Print