Ubiquiti Cloud Key Upgrade – Can’t Login to Web UI

TL;DR:

A recent update to the UCK (Ubiquiti Cloud Key) controller software to add controller version caching caused the device to not start properly. Updating the unifi package via apt and rebooting solved the issue. SSH using the root account and your user password.

apt update
apt upgrade -y
shutdown -r now

—————

I recently upgraded the firmware on my Ubiquiti Cloud Key and was not able to login to the UI this morning. I was able to get to the non-SSL port but was having trouble logging in using any combination of known credentials.

I was able to login via SSH and perform some investigation on the Cloud Key. To login via SSH, use the root account with your account password.

Controller software: v5.11.50
Cloud Key software: v1.1.6

The nginx config for the Web UI indicates the service will listen on all interfaces, using IPv6:

server {
        listen [::]:80 ipv6only=off;
...
server {
        listen [::]:443 ssl ipv6only=off;
        ssl_protocols TLSv1.2;
...

I changed this to use IPv4 only, and restarted nginx:

server {
        # listen [::]:80 ipv6only=off;
        listen *:80;

...
server {
        # listen [::]:443 ssl ipv6only=off;
        listen *:443 ssl;
        ssl_protocols TLSv1.2;

I was still not able to access the Web UI on port 8443, so I checked /etc/ for whatever was listening on that port, that was not working:

root@UniFi-CloudKey:~# egrep -iR 8443 /etc/
/etc/bt-proxy/services.d/unifi.json:    "host": "https://localhost:8443",

It looks like the unifi service is not started. I looked for the right service to restart using systemctl:

root@UniFi-CloudKey:~# systemctl | egrep -i unifi
bt-proxy.service                                                                             loaded active running   UniFi CloudKey Bluetooh API
ubnt-unifi-setup.service                                                                     loaded active exited    Ubiquiti UniFi Setup service
unifi.service                                                                                loaded active running   unifi

I then restarted the service:

sysctmctl restart unifi

That did not work. Even a reboot results in the same behavior.

Maybe we’re out of disk space?

root@UniFi-CloudKey:~# df -h
Filesystem                     Size  Used Avail Use% Mounted on
aufs-root                      2.9G  222M  2.7G   8% /
udev                            10M     0   10M   0% /dev
tmpfs                          404M  444K  404M   1% /run
/dev/disk/by-label/userdata    2.9G  222M  2.7G   8% /mnt/.rwfs
/dev/disk/by-partlabel/rootfs  329M  329M     0 100% /mnt/.rofs
tmpfs                         1009M     0 1009M   0% /dev/shm
tmpfs                          5.0M     0  5.0M   0% /run/lock
tmpfs                         1009M     0 1009M   0% /sys/fs/cgroup
tmpfs                         1009M   64K 1009M   1% /tmp
/dev/mmcblk0p8                  11G  1.1G  9.3G  10% /srv
/dev/mmcblk1p1                 7.2G   17M  7.2G   1% /data

The /dev/disk/by-partlabel/rootfs volume is 100% full, but that might be a read only device with firmware. More to look at there.

In reviewing the filesystem further, it looks like the unifi logs are at /srv/unifi/logs/server.log. This log suggests that the [mongo] database cannot be upgraded to the version requested:

[2019-12-03T09:17:32,155]  WARN  db     - DbServer cannot start due to memory error, restarting with repair
[2019-12-03T09:22:21,198]  ERROR db     - We do not support upgrading from 5.12.35.

Firmware version: UCK.mtk7623.v1.1.6.c289a3c.191031.0856

A little searching for the unsupported upgrade error resulted in finding a thread that suggested upgrading via apt, followed by a reboot:

apt update
apt upgrade -y
shutdown -r now

It looks like the root cause of this issue is that version 1.1.6 of the controller includes caching and this may have caused the break.


Comments

7 responses to “Ubiquiti Cloud Key Upgrade – Can’t Login to Web UI”

  1. Ahmed Nasheed Avatar
    Ahmed Nasheed

    Hi, i was having the same issue. Just googled before i went on troubleshooting and this solved it. Hope you post this in unifi forums.
    Thanks.

  2. Olle Sundblad Avatar
    Olle Sundblad

    One our lost in the Unifi forums and the solution is this simple.

    Super thanks!

  3. Brian Chan Avatar
    Brian Chan

    Exact same thing happened to me. Thank you so much for posting the solution. You’re awesome!

  4. This did the trick. Awesome!
    Thanks a lot.

  5. John Sterbonic Avatar
    John Sterbonic

    You are a heck of a documenter! Thank you so much for putting this out there, the exact same issue with exact same firmware. Using APT worked like a charm, before I found your article I had found the ‘server.log’ file at /srv/unifi/logs and ‘tail server.log’ allowed me to finally get a piece of the error: ERROR DB “We do not support upgrading from 5.12.35.” Without that I might not have found your awesome write up! Thank you sir!

  6. Carl Veee Avatar
    Carl Veee

    Thank you!!!

  7. Great work in solving this, just helped me out, thank you!

Leave a Reply

Your email address will not be published. Required fields are marked *