Constant Server Changes

[mm0opx]

Guys,

This evening (02/01/24), RM noise not usuable as I am seeing drop outs randomly (every 30 - 60 seconds, sometimes longer) and in the CMD window it looks like its looking for a new best fit server??

Theres no packet loss and delay is good as always at 0.2sec. Is there a way we can have a fixed server? I think all is OK my end?

Regards
Colin

[RandyW]

Colin,

I see the issue in the logs. Your client is repeatedly disconnecting/reconnecting it's TCP connection from/to all 3 servers

This is from 1 server:

... logs | grep mm0opx | grep " 20:" | grep reset
2024-01-02 20:06:13.038 listen to client: error: client: AX1 ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:07:11.397 listen to client: error: client: 4AW ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:07:43.548 listen to client: error: client: YRR ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:09:12.065 listen to client: error: client: RA3 ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:10:42.733 listen to client: error: client: LNU ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:11:18.976 listen to client: error: client: K25 ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:12:13.352 listen to client: error: client: MYP ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:12:39.440 listen to client: error: client: DI5 ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:13:43.780 listen to client: error: client: ZKX ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:15:12.427 listen to client: error: client: SG1 ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:16:42.987 listen to client: error: client: USB ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:17:53.421 listen to client: error: client: 9M8 ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:18:13.598 listen to client: error: client: H99 ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:19:42.189 listen to client: error: client: 7LG ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:21:12.923 listen to client: error: client: I99 ( mm0opx ) exception [Errno 104] Connection reset by peer

I see no evidence that this is happening to anyone else:

... logs | grep " 20:1" | grep reset
2024-01-02 20:10:42.733 listen to client: error: client: LNU ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:11:18.976 listen to client: error: client: K25 ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:12:13.352 listen to client: error: client: MYP ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:12:39.440 listen to client: error: client: DI5 ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:13:43.780 listen to client: error: client: ZKX ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:15:12.427 listen to client: error: client: SG1 ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:16:42.987 listen to client: error: client: USB ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:17:53.421 listen to client: error: client: 9M8 ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:18:13.598 listen to client: error: client: H99 ( mm0opx ) exception [Errno 104] Connection reset by peer
2024-01-02 20:19:42.189 listen to client: error: client: 7LG ( mm0opx ) exception [Errno 104] Connection reset by peer

(No other user is in this list during the same time period)


Indeed other users are online, producing normal log data on this server:

... logs | grep " 20:1" | grep Stats
2024-01-02 20:10:10.475 LOG: client: xxx ( myradio ) 2024-01-02 15:10:12.783007 Stats report
2024-01-02 20:10:12.033 LOG: client: xxx ( w7byu ) 2024-01-02 13:10:11.930229 Stats report
2024-01-02 20:10:25.883 LOG: client: xxx ( pd0dh ) 2024-01-02 21:10:25.749030 Stats report
2024-01-02 20:10:40.429 LOG: client: xxx ( 30 cg411 ) 2024-01-02 21:10:41.531735 Stats report
2024-01-02 20:10:41.350 LOG: client: xxx ( 26te555 ) 2024-01-02 20:11:04.093989 Stats report
2024-01-02 20:10:43.158 LOG: client: xxx ( wb5leo ) 2024-01-02 12:10:42.773303 Stats report
2024-01-02 20:10:51.400 LOG: client: xxx ( pa3exc ) 2024-01-02 21:10:51.294882 Stats report
2024-01-02 20:10:51.793 LOG: client: xxx ( n3fl ) 2024-01-02 15:10:51.716789 Stats report
2024-01-02 20:10:53.565 LOG: client: xxx ( ce6tk ) 2024-01-02 17:10:53.389510 Stats report
2024-01-02 20:11:06.552 LOG: client: xxx ( ab7bs ) 2024-01-02 12:11:06.456509 Stats report
2024-01-02 20:11:09.556 LOG: client: xxx ( w7nee ) 2024-01-02 12:11:09.419273 Stats report
2024-01-02 20:11:12.012 LOG: client: xxx ( thijn ) 2024-01-02 21:11:13.221612 Stats report
2024-01-02 20:11:13.322 LOG: client: xxx ( la0ge ) 2024-01-02 21:11:13.159575 Stats report
2024-01-02 20:11:13.326 LOG: client: xxx ( k9gtr ) 2024-01-02 14:11:13.159095 Stats report
2024-01-02 20:11:38.879 LOG: client: xxx ( myradio ) 2024-01-02 15:11:41.217748 Stats report
2024-01-02 20:11:41.554 LOG: client: xxx ( w7byu ) 2024-01-02 13:11:41.511169 Stats report
2024-01-02 20:11:55.206 LOG: client: xxx ( pd0dh ) 2024-01-02 21:11:55.105631 Stats report
2024-01-02 20:12:07.430 LOG: client: xxx ( 30 cg411 ) 2024-01-02 21:12:08.552819 Stats report
2024-01-02 20:12:10.560 LOG: client: xxx ( 26te555 ) 2024-01-02 20:12:33.282105 Stats report
2024-01-02 20:12:13.056 LOG: client: xxx ( wb5leo ) 2024-01-02 12:12:12.639424 Stats report
2024-01-02 20:12:20.385 LOG: client: xxx ( n3fl ) 2024-01-02 15:12:20.297231 Stats report
...
2024-01-02 20:16:59.962 LOG: client: xxx ( w7nee ) 2024-01-02 12:16:59.922402 Stats report
2024-01-02 20:17:04.051 LOG: client: xxx ( ab7bs ) 2024-01-02 12:17:03.948153 Stats report
2024-01-02 20:17:04.845 LOG: client: xxx ( k9gtr ) 2024-01-02 14:17:04.772200 Stats report
2024-01-02 20:17:07.303 LOG: client: xxx ( la0ge ) 2024-01-02 21:17:07.151162 Stats report
2024-01-02 20:17:26.526 LOG: client: xxx ( ct7aov ) 2024-01-02 20:17:26.366660 Stats report
2024-01-02 20:17:31.235 LOG: client: xxx ( myradio ) 2024-01-02 15:17:33.558795 Stats report
2024-01-02 20:17:37.547 LOG: client: xxx ( w7byu ) 2024-01-02 13:17:37.418459 Stats report
2024-01-02 20:17:56.138 LOG: client: xxx ( 30 cg411 ) 2024-01-02 21:17:57.208910 Stats report
2024-01-02 20:18:09.650 LOG: client: xxx ( 26te555 ) 2024-01-02 20:18:32.371950 Stats report
2024-01-02 20:18:12.113 LOG: client: xxx ( wb5leo ) 2024-01-02 12:18:11.706770 Stats report
2024-01-02 20:18:18.064 LOG: client: xxx ( ce6tk ) 2024-01-02 17:18:17.957105 Stats report
2024-01-02 20:18:18.800 LOG: client: xxx ( pa3exc ) 2024-01-02 21:18:18.706291 Stats report
2024-01-02 20:18:27.664 LOG: client: xxx ( w7nee ) 2024-01-02 12:18:27.542225 Stats report
2024-01-02 20:18:28.419 LOG: client: xxx ( ap2fly ) 2024-01-03 01:18:28.237278 Stats report
2024-01-02 20:18:32.955 LOG: client: xxx ( k9gtr ) 2024-01-02 14:18:32.899264 Stats report
2024-01-02 20:18:33.110 LOG: client: xxx ( ab7bs ) 2024-01-02 12:18:33.071711 Stats report
2024-01-02 20:18:36.011 LOG: client: xxx ( la0ge ) 2024-01-02 21:18:35.932081 Stats report
2024-01-02 20:18:55.605 LOG: client: xxx ( ct7aov ) 2024-01-02 20:18:55.450307 Stats report
2024-01-02 20:19:00.207 LOG: client: xxx ( myradio ) 2024-01-02 15:19:02.593364 Stats report
2024-01-02 20:19:06.667 LOG: client: xxx ( w7byu ) 2024-01-02 13:19:06.503335 Stats report
2024-01-02 20:19:23.636 LOG: client: xxx ( 30 cg411 ) 2024-01-02 21:19:24.746581 Stats report
2024-01-02 20:19:38.942 LOG: client: xxx ( 26te555 ) 2024-01-02 20:20:01.636467 Stats report
2024-01-02 20:19:41.518 LOG: client: xxx ( wb5leo ) 2024-01-02 12:19:41.065754 Stats report
2024-01-02 20:19:46.769 LOG: client: xxx ( ce6tk ) 2024-01-02 17:19:46.641471 Stats report
2024-01-02 20:19:47.608 LOG: client: xxx ( pa3exc ) 2024-01-02 21:19:47.525706 Stats report
2024-01-02 20:19:55.265 LOG: client: xxx ( w7nee ) 2024-01-02 12:19:55.153078 Stats report
2024-01-02 20:19:59.655 LOG: client: xxx ( k9gtr ) 2024-01-02 14:19:59.550898 Stats report
[edit] I removed the stream IDs [/edit]

I cannot say what was happening, but it appears isolated to your connection, and I'm not sure what to suggest next.

[mm0opx]

Thanks Randy,

Wonder if anyone else in the UK can test for me. I will ask about. It was the strangest thing. I will try again tonight.

How does the server selection work? If the difference is minimal will it still change? Can the logic be set so it remains on 1 server for a set length of time? Perhaps its set like that already?

Colin

[HB9VQQ]

Hi Colin,

Does the problem still exist? If yes, what if you connect to the internet by another means like GSM/LTE?

How does the server selection work? If the difference is minimal will it still change?

https://ournetplace.com/rm-noise/new-se ... rs-wanted/

how it works.JPG (31.96 KiB) Viewed 3874 times

In the RMN Client log output, you should see something like this:

Code: Select all

2024-01-03 10:10:10.783533 servers and udp: server selection triggered
2024-01-03 10:10:10.783533 server: rmnoise1.rmnoise.com  latency avg/max: 0.163/1.499 -100  lost/sent: 13/263321 (0.005%)  out: 0  -0.2  util: 20.0  score: 1.683
2024-01-03 10:10:10.783533 server: rmnoise2.rmnoise.com  latency avg/max: 0.167/1.429 -100  lost/sent: 48/267308 (0.018%)  out: 0  -2.6  util: 1.2  score: 1.794
2024-01-03 10:10:10.783533 server: rmnoise3.rmnoise.com  latency avg/max: 0.006/1.262 -1.0  lost/sent: 10/231797 (0.004%)  out: 0  -0.0  util: 1.2  score: 88.813

The server with the highest score will be selected and connected. If you have some sort of firewall at the internet perimeter, you can block communication to some RMN servers and make the connection "sticky", although I would not recommend this permanently. If the "sticky" server becomes unavailable for some reason, failover to another server will not work.

The best available server is reselected every 30 minutes by default.

73
Roland

[RandyW]

If the difference is minimal will it still change?

Yes.

I spent a lot of effort to ensure the switch was seamless. Often it happens without loosing a single audio frame. In the case you move from a high latency server to a much lower latency server, there may be a couple of audio frames dropped and reported as "out of order". This cannot be avoided because you have to drop those frames in order to take advantage of the [new] lower latency.

I also use this mechanism to move clients away from a server in order to perform maintenance without a single second of downtime [in a perfect world.]


When you were experiencing problems, did you try reloading the RM Noise client?


To be clear, your issue wasn't related to the server selection process. Your TCP connection was repeatedly broken. After some thought I do have a suggestion. Once the TCP connections were broken, the client cannot send logging to the server. If this occurs again, include your log data during that time period as it will be a superset of my data.


I also examined the code. With the latest software version, there are only 2 reasons the client will break the TCP connection: 1) the window is closed and 2) the connected servers is no longer in the server list provided by DNS

#2 cannot be the reason because it was implemented on later client versions. You are using v0.21.9.7-pre9 which simply did not have have that code.

[mm0opx]

Hi Guys,

Today id the first of me getting back to check. All is good and nothing has changed, always the problem with intermittent faults.

Hopefully it behaves but If it does it again, I will make the suggested checks.

Thanks again

Colin