Ubiquiti Cloud Key Upgrade – Can’t Login to Web UI

TL;DR:

A recent update to the UCK (Ubiquiti Cloud Key) controller software to add controller version caching caused the device to not start properly. Updating the unifi package via apt and rebooting solved the issue. SSH using the root account and your user password.

apt update
apt upgrade -y
shutdown -r now

—————

I recently upgraded the firmware on my Ubiquiti Cloud Key and was not able to login to the UI this morning. I was able to get to the non-SSL port but was having trouble logging in using any combination of known credentials.

I was able to login via SSH and perform some investigation on the Cloud Key. To login via SSH, use the root account with your account password.

Controller software: v5.11.50
Cloud Key software: v1.1.6

The nginx config for the Web UI indicates the service will listen on all interfaces, using IPv6:

server {
        listen [::]:80 ipv6only=off;
...
server {
        listen [::]:443 ssl ipv6only=off;
        ssl_protocols TLSv1.2;
...

I changed this to use IPv4 only, and restarted nginx:

server {
        # listen [::]:80 ipv6only=off;
        listen *:80;

...
server {
        # listen [::]:443 ssl ipv6only=off;
        listen *:443 ssl;
        ssl_protocols TLSv1.2;

I was still not able to access the Web UI on port 8443, so I checked /etc/ for whatever was listening on that port, that was not working:

root@UniFi-CloudKey:~# egrep -iR 8443 /etc/
/etc/bt-proxy/services.d/unifi.json:    "host": "https://localhost:8443",

It looks like the unifi service is not started. I looked for the right service to restart using systemctl:

root@UniFi-CloudKey:~# systemctl | egrep -i unifi
bt-proxy.service                                                                             loaded active running   UniFi CloudKey Bluetooh API
ubnt-unifi-setup.service                                                                     loaded active exited    Ubiquiti UniFi Setup service
unifi.service                                                                                loaded active running   unifi

I then restarted the service:

sysctmctl restart unifi

That did not work. Even a reboot results in the same behavior.

Maybe we’re out of disk space?

root@UniFi-CloudKey:~# df -h
Filesystem                     Size  Used Avail Use% Mounted on
aufs-root                      2.9G  222M  2.7G   8% /
udev                            10M     0   10M   0% /dev
tmpfs                          404M  444K  404M   1% /run
/dev/disk/by-label/userdata    2.9G  222M  2.7G   8% /mnt/.rwfs
/dev/disk/by-partlabel/rootfs  329M  329M     0 100% /mnt/.rofs
tmpfs                         1009M     0 1009M   0% /dev/shm
tmpfs                          5.0M     0  5.0M   0% /run/lock
tmpfs                         1009M     0 1009M   0% /sys/fs/cgroup
tmpfs                         1009M   64K 1009M   1% /tmp
/dev/mmcblk0p8                  11G  1.1G  9.3G  10% /srv
/dev/mmcblk1p1                 7.2G   17M  7.2G   1% /data

The /dev/disk/by-partlabel/rootfs volume is 100% full, but that might be a read only device with firmware. More to look at there.

In reviewing the filesystem further, it looks like the unifi logs are at /srv/unifi/logs/server.log. This log suggests that the [mongo] database cannot be upgraded to the version requested:

[2019-12-03T09:17:32,155]  WARN  db     - DbServer cannot start due to memory error, restarting with repair
[2019-12-03T09:22:21,198]  ERROR db     - We do not support upgrading from 5.12.35.

Firmware version: UCK.mtk7623.v1.1.6.c289a3c.191031.0856

A little searching for the unsupported upgrade error resulted in finding a thread that suggested upgrading via apt, followed by a reboot:

apt update
apt upgrade -y
shutdown -r now

It looks like the root cause of this issue is that version 1.1.6 of the controller includes caching and this may have caused the break.