Hot Topic (More than 10 Replies) Sesame server does not recover? (Read 2956 times)
jacker
Member
*
Offline



Posts: 42
Location: Milwaukee, WI
Joined: May 20th, 2008
Sesame server does not recover?
Jan 5th, 2017 at 5:24pm
Print Post Print Post  
I have an issue with 2.6.2. I am running a client-server model. 18 licenses. This morning I was notified that no one could connect. I remoted into the server and noticed all 18 licenses were consumed mostly by one user who apparently had tried to connect over and over again.

So I figured, "Easy fix, I'll just kill all of his client processes."

I did that...I logged into the client's PC, killed his Sesame client processes and brought the connections down to about 6 on the server. But still, no one could connect! Even myself so I couldn't even enter Server Administrator and do an orderly shutdown. My only option was to risk database corruption and do an Emergency Shutdown and then unlock all the applications.

This is hardly an improvement over 2.5!! Sesame is no longer self correcting. Now it will require someone who has access to the server (myself) to take action!

Guess my vacation plans are cancelled.   Angry
  

Jack - I'm smokin' past the filter and it's burnin' my lips...
Back to top
YIM YIM  
IP Logged
 
Ray the Reaper
Global Moderator
Members
Lantica Support
*****
Offline


The One & The Only

Posts: 2480
Joined: Aug 20th, 2003
Re: Sesame server does not recover?
Reply #1 - Jan 5th, 2017 at 5:44pm
Print Post Print Post  
jacker wrote on Jan 5th, 2017 at 5:24pm:
This is hardly an improvement over 2.5!! Sesame is no longer self correcting. Now it will require someone who has access to the server (myself) to take action!


It is self correcting in regards to network communication in the same way that 2.5 was, and has even more network self correcting(It can reconnect after a drop in network communication)  You'll see the client connection count decrease after a short amount of time(<2 minutes) after killing a client process.

A few questions that may narrow down what was happening:
Was the Sesame process on the Server computer using CPU and Memory at the time that no body could connect?
Was the Sesame Server processing commands? This can be seen on the Server window.

-Ray
  

Raymond Yoxall Consulting
ray.yoxall@gmail.com
ryoxall@lantica.com
Sesame Applications, Design and Support
Back to top
IP Logged
 
jacker
Member
*
Offline



Posts: 42
Location: Milwaukee, WI
Joined: May 20th, 2008
Re: Sesame server does not recover?
Reply #2 - Jan 19th, 2017 at 6:06pm
Print Post Print Post  
This is happening frequently, at least once a week...it's occurring right now.

Something causes clients to stop connecting, possibly too many people trying to connect at once...however, once it happens, my only option is to do an emergency shutdown...even if the number of clients drops below the number of licenses. Each client will sit on the little yellow connect window with "Connecting 60" (or 59) showing til you kill it manually.

I've been waiting for my client to connect for 20 minutes now, even tho the server tells me there are at least 3 slots open, sometimes more...

  

Sesame-Capture01.JPG ( 195 KB | 162 Downloads )
Sesame-Capture01.JPG

Jack - I'm smokin' past the filter and it's burnin' my lips...
Back to top
YIM YIM  
IP Logged
 
Ray the Reaper
Global Moderator
Members
Lantica Support
*****
Offline


The One & The Only

Posts: 2480
Joined: Aug 20th, 2003
Re: Sesame server does not recover?
Reply #3 - Jan 20th, 2017 at 4:45pm
Print Post Print Post  
The clients that are currently connected, are they still able to function?

What are the currently connected clients doing when this happens? If they are running a large routine, does it continue to run?

Does the commands processed(the first number in that section) change when a new client attempts to connect? If so by how many?

On the Server computer, when this happens if you bring up a command prompt and run a NetStat command(Netstat /a /b /p tcp will give you all the TCP ports open and what program opened them) are both communication ports(20,000 and 20,001 by default) open on the Sesame Server? Are there two communication ports open to each Sesame client as well?

In the disconnected client list, does one computer show up over and over again? That indicates a communication issue with that particular client.

-Ray
  

Raymond Yoxall Consulting
ray.yoxall@gmail.com
ryoxall@lantica.com
Sesame Applications, Design and Support
Back to top
IP Logged
 
jacker
Member
*
Offline



Posts: 42
Location: Milwaukee, WI
Joined: May 20th, 2008
Re: Sesame server does not recover?
Reply #4 - Jan 23rd, 2017 at 2:29pm
Print Post Print Post  
Monday morning and Sesame is dead in the water once again...to answer Ray's questions:

The clients that are currently connected, are they still able to function?
No.

What are the currently connected clients doing when this happens? If they are running a large routine, does it continue to run?
Nothing special, the Sesame apps are very simple. No one can function, commands stop processing.

Does the commands processed(the first number in that section) change when a new client attempts to connect? If so by how many?
No.

On the Server computer, when this happens if you bring up a command prompt and run a NetStat command(Netstat /a /b /p tcp will give you all the TCP ports open and what program opened them) are both communication ports(20,000 and 20,001 by default) open on the Sesame Server? Are there two communication ports open to each Sesame client as well?
Yes and Yes.


In the disconnected client list, does one computer show up over and over again? That indicates a communication issue with that particular client.
They all show up eventually it seems.

Today I'm getting some color!


  

Sesame-Capture04a.jpg ( 131 KB | 157 Downloads )
Sesame-Capture04a.jpg

Jack - I'm smokin' past the filter and it's burnin' my lips...
Back to top
YIM YIM  
IP Logged
 
jacker
Member
*
Offline



Posts: 42
Location: Milwaukee, WI
Joined: May 20th, 2008
Re: Sesame server does not recover?
Reply #5 - Jan 23rd, 2017 at 3:30pm
Print Post Print Post  
Just did an Emergency Shutdown since I couldn't connect either...unlocked all the apps and dbs.

One office is still having issues, getting kicked off and having a hard time getting connected. They are the ones showing up in red:



  

Sesame-Capture07a.jpg ( 132 KB | 169 Downloads )
Sesame-Capture07a.jpg

Jack - I'm smokin' past the filter and it's burnin' my lips...
Back to top
YIM YIM  
IP Logged
 
jacker
Member
*
Offline



Posts: 42
Location: Milwaukee, WI
Joined: May 20th, 2008
Re: Sesame server does not recover?
Reply #6 - Jan 23rd, 2017 at 5:59pm
Print Post Print Post  
Even when I try to do an orderly shutdown it doesn't work!

I had everyone disconnect, then I went into Server Admin and requested a Stop Server...I watched the countdown and it kept flashing "3 remaining clients" even though I was the only one...when the countdown completed it does nothing.

  

Sesame-Capture08a.jpg ( 112 KB | 158 Downloads )
Sesame-Capture08a.jpg

Jack - I'm smokin' past the filter and it's burnin' my lips...
Back to top
YIM YIM  
IP Logged
 
jacker
Member
*
Offline



Posts: 42
Location: Milwaukee, WI
Joined: May 20th, 2008
Re: Sesame server does not recover?
Reply #7 - Jan 23rd, 2017 at 6:00pm
Print Post Print Post  
Stop Server doesn't seem to work...
  

Sesame-Capture09a.jpg ( 154 KB | 169 Downloads )
Sesame-Capture09a.jpg

Jack - I'm smokin' past the filter and it's burnin' my lips...
Back to top
YIM YIM  
IP Logged
 
Ray the Reaper
Global Moderator
Members
Lantica Support
*****
Offline


The One & The Only

Posts: 2480
Joined: Aug 20th, 2003
Re: Sesame server does not recover?
Reply #8 - Jan 25th, 2017 at 4:34pm
Print Post Print Post  
Uncheck the Notify All Clients button and Sesame Server will shutdown. You are having communication issues on the Notify Port so Sesame is trying to Notify clients that it is having trouble communicating with.

Looking at those screenshots either one or both of the communication ports are being closed from your Sesame Server to the clients in yellow and red, or a communication between the Server and the Client is being directed elsewhere/lost. It could be firewall, bad or unhappy NIC, bad switch/hub. If you check Netstat on the server computer when you have red clients you will see if a port is being closed. TCPView is also a free utility that shows when ports are opened and closed in Windows, if you prefer a GUI over a command prompt. It can also be set to auto update.

Does it always happen on Mondays or in the morning? If so what tasks are scheduled to run over the weekend/night?

-Ray
  

Raymond Yoxall Consulting
ray.yoxall@gmail.com
ryoxall@lantica.com
Sesame Applications, Design and Support
Back to top
IP Logged
 
jacker
Member
*
Offline



Posts: 42
Location: Milwaukee, WI
Joined: May 20th, 2008
Re: Sesame server does not recover?
Reply #9 - Jan 25th, 2017 at 5:16pm
Print Post Print Post  
It happens often on Monday mornings, especially if clients forget to exit Sesame over the weekend...the apps themselves are very simple, just adding or updating records manually, no batch processing or fancy subroutines.

The server is a commercial cloud server with the Sesame ports whitelisted on the firewall, so it should be rock solid. These issues are all intermittent, Sesame runs fine most of the time.

One definite issue is when a user can't connect and launches Sesame over and over and does not realize that multiple processes are running in the background. Once the license limit is reached, corruption occurs and EVERYONE will no longer be able to use Sesame or even connect. Even if everyone disconnects, at this point Sesame is dead in the water. I have to manually shut down and restart the server, and unlock all the databases in order for anyone to proceed normally. I have to do this about 3 or 4 times a week.
  

Jack - I'm smokin' past the filter and it's burnin' my lips...
Back to top
YIM YIM  
IP Logged
 
Ray the Reaper
Global Moderator
Members
Lantica Support
*****
Offline


The One & The Only

Posts: 2480
Joined: Aug 20th, 2003
Re: Sesame server does not recover?
Reply #10 - Jan 26th, 2017 at 5:46pm
Print Post Print Post  
Are they any scheduled tasks in Windows that run over the weekend?

I believe the issue has already happened by the time that the user attempts to launch multiple clients. By that point something has already happened with the communication ports on the server. On the server computer in the C:\Sesame2 folder there is a file named s_log0000.log, send that file to support@lantica.com and I'll see what it has to say.

-Ray
  

Raymond Yoxall Consulting
ray.yoxall@gmail.com
ryoxall@lantica.com
Sesame Applications, Design and Support
Back to top
IP Logged