A Design Scheme for Personal Cloud

Posted on 2021-10-27 Edited on 2023-01-27 In Geek

21 century is a data world, we generate data, we consume data, and we protect data. Want to do more? Maybe you can follow this article.

Why

Nowdays, our personal data is growing bigger and bigger. Photos, videos, documents, all this kind of files fill in the computer, smart phone or external drive.

I can’t say they are all important for me, but some does mean a lot and I don’t want to lose them.

Meanwhile, our parents also generate many data exeryday, but they don’t even have any backup, if their mobile phone gets broken, they will lose everything.

What’s worse, the public cloud service is too expensive for normal people, you knwo we just want to store some photos, not something like the password of nuclear bomb. And the privacy is also a disputed issue.

Then I came up with the idea of building up a personal cloud service, it should be cheap and kind of reliable. Also, it would be better if we can use open source softwares.

I hope this article could help you.

Scheme

As a cloud, it must can be accessed remotely, thus we need a public IP address. And of course, to keep private, https will be considered, then a domain is needed as well.

To control the budget, it’s not acceptable to purchase a VPS with large storage capacity, a NAS would be a good choice. But the retail NAS is also too expensive, so I will make up one by myself, after all, it’s just a computer basically. Because the NAS is in the private network, some technology will be used to make it accessible.

After days of investigation, I got the following list:

Hardware

CPU: Intel J3455
Motherboard: ASRock J3455-ITX (CPU included)
Memory: SK Hynix 8GB DDR3L dual channel
System drive: Intel 520 SSD 120GB
Data drive: WD Purple WD40EJRX 4TB * 2
Case: U-NAS NSC-201

Software

OS: Ubuntu Server 20.04
Cloud Service: Nextcloud
Reverse proxy: FRP & Nginx
Filesystem: ZFS

VPS

Aliyun Simple Application Server

The whole scheme is like this:

Storage

Though budget matters, but we don’t want to lose data anyway. And the most dangerous situation is hardware broken, especially hard driver failure. To mitigate this condition, we will use a software RAID1, and it’s achieved by a mirror pool of ZFS.

To make use of ZFS on Ubuntu, we need to install it first:

1	sudo apt install zfsutils-linux

Then, suppose the data driver is /dev/sdb and /dev/sdc, to create a mirror pool named storage:

1
2
3

sudo zpool create storage mirror /dev/sdb /dev/sdc
sudo zfs set mountpoint=none storage
sudo zpool status

Finally we create a ZFS filesystem for Nextcloud and mount it to /var/www/nextcloud:

1 2	sudo zfs create storage/nextcloud sudo zfs set mountpoint=/var/www/nextcloud storage/nextcloud

Nextcloud

Installation

We will install Nextcloud on Ubuntu Server 20.04.

Install packages

1	sudo apt install apache2 mariadb-server libapache2-mod-php7.4 php7.4-gd php7.4-mysql php7.4-curl php7.4-mbstring php7.4-intl php7.4-gmp php7.4-bcmath php-imagick php7.4-xml php7.4-zip

Install Nextcloud

Goto Nextcloud, select Download for server and copy the link, then download and unpack the tarball, copy the nextcloud directory to webroot. We need to change the owner as well to make Apache be able to modify the files in it.
1
2
3
4
wget https://download.nextcloud.com/server/releases/nextcloud-22.2.0.tar.bz2
tar -xf nextcloud-22.2.0.tar.bz2
sudo cp -r nextcloud /var/www/
sudo chown -R www-data:www-data /var/www/nextcloud/

Configure database

We will create a user as well as a database for Nextcloud. Please replace the username and password to the one you want.

sudo mysql

CREATE USER 'username'@'localhost' IDENTIFIED BY 'password';
CREATE DATABASE IF NOT EXISTS nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
GRANT ALL PRIVILEGES ON nextcloud.* TO 'username'@'localhost';
FLUSH PRIVILEGES;

quit;

Configure Apache

For security reasons, https will be used, thus we need to generate a self-signed certificate firstly.

1
2
3

openssl req -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out nextcloud.crt -keyout nextcloud.key
sudo mv nextcloud.crt /etc/ssl/certs/
sudo mv nextcloud.key /etc/ssl/private/

Then create a new Apache config file for Nextcloud: /etc/apache2/sites-available/nextcloud.conf, with the following content:

<IfModule mod_ssl.c>
        <VirtualHost _default_:443>
                ServerName nextcloud.colorfulshark.net

                <IfModule mod_headers.c>
                        Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains"
                </IfModule>

                DocumentRoot /var/www/nextcloud

                ErrorLog ${APACHE_LOG_DIR}/error.log
                CustomLog ${APACHE_LOG_DIR}/access.log combined

                SSLEngine on

                SSLCertificateFile      /etc/ssl/certs/nextcloud.crt
                SSLCertificateKeyFile /etc/ssl/private/nextcloud.key

                <FilesMatch "\.(cgi|shtml|phtml|php)$">
                                SSLOptions +StdEnvVars
                </FilesMatch>
                <Directory /usr/lib/cgi-bin>
                                SSLOptions +StdEnvVars
                </Directory>
        </VirtualHost>
</IfModule>

Enable necessary Apache modules

1	sudo a2enmod rewrite headers env dir mime ssl

Enable Nextcloud website

1	sudo a2ensite nextcloud.conf

Reload Apache service

1	sudo systemctl restart apache2

Configure Nextcloud

Now we can access the Nextcloud website in Chrome, because we are using self-signed certificate, the browser may refuse to open the website, we can simply type thisisunsafe on the blocked page.😂
1
https://10.0.0.10
On this page, we need to input following information:
- admin username
- admin password
- data folder
- mysql username
- mysql password
- mysql database name
Finally, a welcome page will show up.

Optimization

Increate PHP memory limitation

sudo vim /etc/php/7.4/apache2/php.ini
<<<
memory_limit = 512M
>>>
sudo systemctl restart apache2

Disable PHP output_buffering

sudo vim /etc/php/7.4/apache2/php.ini
<<<
output_buffering = Off
>>>
sudo systemctl restart apache2

Set AllowOverride to All for /var/www/ to protect data

sudo vim /etc/apache2/apache2.conf
<<<
<Directory /var/www/>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
</Directory>
>>>
sudo systemctl restart apache2

Enable HTTP Strict Transport Security

<VirtualHost *:443>
  ServerName cloud.nextcloud.com
    <IfModule mod_headers.c>
      Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains"
    </IfModule>
 </VirtualHost>

Enable SVG support for php-imagick
1
sudo apt install imagemagick

Enable Memory caching

sudo apt install php-apcu
sudo vim /var/www/nextcloud/config/config.php
<<<
'memcache.local' => '\OC\Memcache\APCu',
>>>
sudo bash -c "echo 'apc.enable_cli = 1' > /etc/php/7.4/cli/conf.d/enable-apc-cli.ini"
sudo systemctl restart apache2

Setup phone region

sudo vim /var/www/nextcloud/config/config.php
<<<
'default_phone_region' => 'CN',
>>>

Pretty URLs

vim /var/www/nextcloud/config/config.php
<<<
'overwrite.cli.url' => 'https://nextcloud.colorfulshark.net',
'htaccess.RewriteBase' => '/',
>>>
sudo -u www-data php /var/www/nextcloud/occ maintenance:update:htaccess

Setup Cron background job

1
2
3

sudo crontab -u www-data -e

*/5  *  *  *  * php -f /var/www/nextcloud/cron.php

Network

This would be an interesting part, because we are going to make the Nextcloud be accessible remotely.

Domain

To make use of https, we need a domain and set a A record to the IP address of VPS. Suppose the domain is nextcloud.colorfulshark.net.

A certificate is also needed for https usage, it can be registered from Letsencrypt.

sudo su && cd
apt install openssl cron socat curl
curl https://get.acme.sh | sh
source ~/.bashrc
~/.acme.sh/acme.sh --register-account -m colorfulshark@gmail.com

domain=nextcloud.colorfulshark.net
cert=/usr/local/etc/certs/nextcloud.colorfulshark.net.cert
key=/usr/local/etc/certs/nextcloud.colorfulshark.net.key
~/.acme.sh/acme.sh --issue -d $domain --standalone --keylength ec-256 --force
~/.acme.sh/acme.sh --installcert -d $domain --ecc --fullchain-file $cert --key-file $key

FRP

Because the NAS is in internal network, to access it remotely, we need to setup reverse proxy for Nextcloud web service by using FRP.

VPS side: /etc/frp/nextcloud.ini

[common]
bind_addr = 0.0.0.0
proxy_bind_addr = 127.0.0.1
bind_port = 7000
vhost_https_port = 7443
token = password
tls_enable = true

NAS side: /etc/frp/nextcloud.ini

[common]
server_addr = nextcloud.colorfulshark.net
server_port = 7000
token = password
tls_only = true
tls_enable = true

[nextcloud]
type = https
local_port = 443
custom_domains = nextcloud.colorfulshark.net

Then accessing localhost:7443 on VPS is equal to accessing localhost:443 on NAS.

Nginx

Though FRP supports TLS for https reverse proxy, Nginx would be better. Because there may exist many web services on VPS and they all use https. By using Nginx, we can proxy pass these services by identifying domain and path.

VPS side

/etc/nginx/sites-available/nextcloud.colorfulshark.net

server {
  listen 443 ssl;
  listen [::]:443 ssl;

  ssl_certificate       /usr/local/etc/certs/nextcloud.colorfulshark.net.cert;
  ssl_certificate_key   /usr/local/etc/certs/nextcloud.colorfulshark.net.key;
  ssl_session_timeout 1d;
  ssl_session_cache shared:MozSSL:10m;
  ssl_session_tickets off;

  ssl_protocols         TLSv1.3;
  ssl_ciphers           ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
  ssl_prefer_server_ciphers off;

  server_name           nextcloud.colorfulshark.net;

  location / {
    proxy_ssl_server_name on;
    proxy_ssl_name $host;
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass https://127.0.0.1:7443;
    client_max_body_size 1024m;
  }
}

Enable this site and resart Nginx

1 2	sudo ln -s /etc/nginx/sites-available/nextcloud.colorfulshark.net /etc/nginx/sites-enabled/ sudo systemctl restart nginx

Nextcloud

Yes, we are back to Nextcloud again, because Nextcloud only accepts the request from trusted domains, thus we need to let it trust nextcloud.colorfulshark.net as well.

Nas side

/var/www/nextcloud/config/config.php

'trusted_domains' =>
array (
  0 => '10.0.0.10',
  1 => 'nextcloud.colorfulshark.net',
),

Now you can assess the Nextcloud through https://nextcloud.colorfulshark.net.

Maintenance

Upgrade

1
2
3

sudo -u www-data php /var/www/nextcloud/updater/updater.phar
sudo -u www-data php /var/www/nextcloud/occ upgrade
sudo -u www-data php /var/www/nextcloud/occ maintenance:mode --off

Backup database

#!/bin/bash

number=$(($(date +%u) % 2))
db_file="nextcloud_${number}.sql"

# enter maintenance mode
sudo -u www-data php --define /var/www/nextcloud/occ maintenance:mode --on

mysqldump --single-transaction --default-character-set=utf8mb4 nextcloud > /var/www/nextcloud/backup/$db_file
sync

# exit maintenance mode
sudo -u www-data php --define /var/www/nextcloud/occ maintenance:mode --off

And set a cron job

1
2
3

sudo crontab -e

0 5 * * * /var/www/nextcloud/backup/backup_db.sh

Restore database

1
2
3

sudo mysql -e "DROP DATABASE nextcloud"
sudo mysql -e "CREATE DATABASE nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci"
sudo mysql nextcloud < nextcloud_0.sql

Restore from system reinstallation
1. Install all necessary packages
2. Restore nextcloud database
3. Create database user and grant permission
4. Finish all configuration and optimization