Page 1 of 1

SSH stopped working on CCR1072

Posted: Wed Dec 30, 2015 2:06 pm
by serjrd
Hi,

For some weird reason SSH server stopped working.
I was unable to find any relevant alerts in logs.
Disabling/enabling SSH service does not help.

Web interface is accessible.
OS version is 6.33.3.

Any ideas?

Re: SSH stopped working on CCR1072

Posted: Wed Dec 30, 2015 2:54 pm
by simenskaa
Try changing from port 22 to something else? What about the other services, do they work?

Re: SSH stopped working on CCR1072

Posted: Wed Dec 30, 2015 4:58 pm
by gtj
If you enabled strong encryption or regenerated the host keys, you may have to reboot.

Check the keys associated with the user.

What happens on the client side with 'ssh -v'?

Re: SSH stopped working on CCR1072

Posted: Wed Dec 30, 2015 5:23 pm
by serjrd
Sorry, I had to reboot the CCR before I had a chance to actually run a sniffer or 'ssh -v', since a lot of script logic was tied to being able to SSH to a device. The service failure happened completely on its own early in the morning and had nothing to do with any key regeneration or anything like that.

Rebooting did revive the SSH server. But this is definitely a serious issue for us.

Re: SSH stopped working on CCR1072

Posted: Mon Jan 04, 2016 11:14 am
by serjrd
The SSH problem occurred once again.

Here's the output of ssh -v and tcpdump:
% ssh -v admin@10.1.1.1                                                                                                                                  130 ↵ [12:10:13]
OpenSSH_6.9p1 Ubuntu-2, OpenSSL 1.0.2d 9 Jul 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 10.1.1.1 [10.1.1.1] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/serjrd/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/serjrd/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/serjrd/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/serjrd/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/serjrd/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/serjrd/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/serjrd/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/serjrd/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9p1 Ubuntu-2
# tcpdump -vv -pi enp3s0 tcp port 22 and ip host 10.1.1.1                                                                                                        [12:09:59]
tcpdump: listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:10:13.496612 IP (tos 0x0, ttl 64, id 3988, offset 0, flags [DF], proto TCP (6), length 52)
    terminal.host.ru.41174 > 10.1.1.1.ssh: Flags [F.], cksum 0xb91f (incorrect -> 0x0ad1), seq 539035462, ack 3432229159, win 229, options [nop,nop,TS val 87825640 ecr 41045864], length 0
12:10:13.535203 IP (tos 0x0, ttl 64, id 58960, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.1.1.ssh > terminal.host.ru.41174: Flags [.], cksum 0x032b (correct), seq 1, ack 1, win 114, options [nop,nop,TS val 41047937 ecr 87825640], length 0
12:10:14.735560 IP (tos 0x0, ttl 64, id 3544, offset 0, flags [DF], proto TCP (6), length 60)
    terminal.host.ru.41216 > 10.1.1.1.ssh: Flags [S], cksum 0xb927 (incorrect -> 0xb58a), seq 2402620358, win 29200, options [mss 1460,sackOK,TS val 87825950 ecr 0,nop,wscale 7], length 0
12:10:14.736366 IP (tos 0x0, ttl 64, id 16624, offset 0, flags [DF], proto TCP (6), length 64)
    10.1.1.1.ssh > terminal.host.ru.41216: Flags [.], cksum 0x4487 (correct), seq 2746626713, ack 3259690751, win 114, options [nop,nop,TS val 41048057 ecr 78017785,nop,nop,sack 1 {3437896904:3437896905}], length 0
12:10:14.736393 IP (tos 0x0, ttl 64, id 3445, offset 0, flags [DF], proto TCP (6), length 40)
    terminal.host.ru.41216 > 10.1.1.1.ssh: Flags [R], cksum 0xa486 (correct), seq 3259690751, win 0, length 0
12:10:15.733545 IP (tos 0x0, ttl 64, id 3545, offset 0, flags [DF], proto TCP (6), length 60)
    terminal.host.ru.41216 > 10.1.1.1.ssh: Flags [S], cksum 0xb927 (incorrect -> 0xb490), seq 2402620358, win 29200, options [mss 1460,sackOK,TS val 87826200 ecr 0,nop,wscale 7], length 0
12:10:15.733891 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.1.1.ssh > terminal.host.ru.41216: Flags [S.], cksum 0x2a13 (correct), seq 1922168460, ack 2402620359, win 14480, options [mss 1460,sackOK,TS val 41048156 ecr 87826200,nop,wscale 7], length 0
12:10:15.733937 IP (tos 0x0, ttl 64, id 3546, offset 0, flags [DF], proto TCP (6), length 52)
    terminal.host.ru.41216 > 10.1.1.1.ssh: Flags [.], cksum 0xb91f (incorrect -> 0x908a), seq 1, ack 1, win 229, options [nop,nop,TS val 87826200 ecr 41048156], length 0
12:10:15.734539 IP (tos 0x0, ttl 64, id 3547, offset 0, flags [DF], proto TCP (6), length 84)
    terminal.host.ru.41216 > 10.1.1.1.ssh: Flags [P.], cksum 0xb93f (incorrect -> 0x27b4), seq 1:33, ack 1, win 229, options [nop,nop,TS val 87826200 ecr 41048156], length 32
12:10:15.734939 IP (tos 0x0, ttl 64, id 44752, offset 0, flags [DF], proto TCP (6), length 52)
    10.1.1.1.ssh > terminal.host.ru.41216: Flags [.], cksum 0x90dc (correct), seq 1, ack 33, win 114, options [nop,nop,TS val 41048157 ecr 87826200], length 0
^C

Re: SSH stopped working on CCR1072

Posted: Mon Jan 04, 2016 11:53 am
by marrold
I've had a similar issue in the past on older ROS versions. I also observed when SSH stopped working, the following command would hang and eventually timeout and suggest sending a SUPOUT to support
/ ip ssh print