Cloud

Table of Contents

A cloud used to be floating in the sky, then it was a bubbly thing on a network diagram where weird things happen. Now a cloud just means "Somebody else's computer.". Of course you get to it over a network and can share things, like files (pictures, music, documents), contacts, calendars and chat, among other things.

For me it means I can get to my stuff no matter where I am; and my stuff is private unless I share it. So I keep it in my home, under lock and key. To share I have a few open ports, encrypted network traffic, and some protected disk arrays with timely backups.

If this sounds interesting so far, here are some contenders for the job:

  • SyncThing [1]
  • Warpinator [2]
  • Cryptomater [3]
  • NextCloud [4]
  1. https://syncthing.net/
  2. https://github.com/linuxmint/warpinator
  3. https://cryptomator.org/
  4. https://nextcloud.com/

SyncThing

Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers in real time.

It is a simple install supporting many operating systems [1]. Debian Ubuntu has a package [2].

The getting started [3] page has very nice instructions.

My experience was very easy to install and syncing was quick and reliable. However I limited my exposure to internal network machine syncs, so no messing with my firewall or port forwarding [4]. If you just want to sync files for a project, picture albums or music collection to a backup host in your home, it should work very well. I had no problems.

They web page states that "All communication is secured using TLS." and commercial support is available [5].

  1. https://syncthing.net/downloads/
  2. https://apt.syncthing.net/
  3. https://docs.syncthing.net/intro/getting-started.html
  4. https://docs.syncthing.net/users/firewall.html#firewall-setup
  5. https://www.kastelo.net/stes/

Warpinator

Warpinator is built to share files across the LAN. I have heard several people and magazines praise how good it works. It is built with Python and available on Github, supporting several operating systems. On Linux Mint it is a simple apt install, probably because it is published on the github linuxmint repository.

The project supplies firewall instructions so it should be able to run remotely if desired. It uses a shared code to secure comminucation. It is un-clear to me if it uses SSL/TLS encryption, and the support would be through Github.

I have not tried it, but it sounds very reasonable for syncing things on a local network since SSL/TLS is not mentioned.

Other platforms include:

Cryptomater

Cryptomator encrypts your data quickly and easily. Afterwards you upload them protected to your favorite cloud service.

The work flow here is to take files from one directory and copy them to another, then encrypting the new directory in preperation of uploading to a cloud service on someone elses computer.

My experience was quite good with it for saving a copy of my development project off-site to DropBox. I would prepare my files into a DropBox directory with DropBox turned off, then turn DropBox on, watch it sync, then turn DropBox off. I was assured that no one outside my network could read my secret project files. Very nice, cheap and easy.

Github reports it is 93% Java, so be aware of that. My usage was all MacOS, so I didn't even realize the Java dependency, but the Linux dependency lists Oracle JDK 16 [3].

They offer enterprise support [1] and also Github [2] bug reporting.

  1. https://cryptomator.org/enterprise/
  2. https://github.com/cryptomator
  3. https://www.oracle.com/java/technologies/javase/products-doc-jdk16certconfig.html

NextCloud

NextCloud allows file storage on another computer, in my house, so that I can easily access these files from almost any other computer/phone/tablet that has authority. It also supports a shared list of Contacts and Calendar. There is a chat function (Talk) to type messages, or voice, or video between two or more other computers.

The app store reports many other apps available as well.

My experience started with OwnCloud, which NextCloud was forked from, and used to have many problems with every new release especially supporting a PostgreSQL database. After OwnCloud went with a entire new infastructure with no more PHP, I decided to try NextCloud, and have been happy ever since. NextCloud is very stable, new releases are smooth, and the many apps seem to work just fine.

Being based on PHP, NextCloud runs better on Apache in my opinion, since nginx does not default with PHP support. The PostgreSQL database support has not had any problems, and the whole stack runs on a small BeagleBone AI (32-bit) or AI-64 (64-bit) SBC, so the Rasberry PI line should also work.

One note, after changing from Debian to RedHat, the php support for Apache (httpd on RedHat), uses php-frm, the PHP FastCGI Process Manager. It seems to work quite well, but be aware there is a systemctl service called php-fpm.

sudo systemctl status php-fpm
● php-fpm.service - The PHP FastCGI Process Manager
     Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled; preset: disabled)
     Active: active (running) since Tue 2023-07-25 22:57:34 EDT; 1 day 9h ago
   Main PID: 940 (php-fpm)
     Status: "Processes active: 0, idle: 13, Requests: 47239, slow: 0, Traffic: 0.2req/sec"
      Tasks: 14 (limit: 48814)
     Memory: 431.1M
        CPU: 43min 15.096s
     CGroup: /system.slice/php-fpm.service
             ├─   940 "php-fpm: master process (/etc/php-fpm.conf)"
             ├─  2642 "php-fpm: pool www"
             ├─  2643 "php-fpm: pool www"
             ├─  2644 "php-fpm: pool www"
             ├─  2645 "php-fpm: pool www"
             ├─  2646 "php-fpm: pool www"
             ├─  2765 "php-fpm: pool www"
             ├─  2767 "php-fpm: pool www"
             ├─  3834 "php-fpm: pool www"
             ├─ 23767 "php-fpm: pool www"
             ├─ 60012 "php-fpm: pool www"
             ├─ 61859 "php-fpm: pool www"
             ├─ 67881 "php-fpm: pool www"
             └─120219 "php-fpm: pool www"

Services

These are services I have/had used without problem:

  • Audio Player for playing music collection, streaming, and playlists
  • Pictures for organizing photographs by year, then topic
  • Contacts for syncing e-mail Thunderbird and iOS
  • Calendar for syncing appointments Thunderbird and iOS
  • Files using NextCloud Clients for Mac, iOS
  • Talk can Send Messages to a phone from Command Line, used for HomeAssistant alerts and run via File: ~/matrix/sendmatrix.sh. Used for monitoring door and window alarms.
IOS App - NextCloud Talk

IMG_84D17B7CC129-1 (2).jpeg

NextCloud Talk uses the Matrix service to detect changes to be sent.

Install Documentation

First install the database;

https://www.postgresql.org/download/

Then apache2;

https://httpd.apache.org/docs/current/install.html

Then nextcloud;

https://nextcloud.com/install/#instructions-server

Then nextcloud apps;

  • contacts
  • calendar
  • talk
  • mattermost

After Nextcloud 27, Talk Mattermost app does not work, and is not needed to send messages through the command line.

Install Steps

What follows are my steps to install NextCloud the way I want it, manually mostly.

It may not be necessary, but it does provide details that might otherwise be missed about how NextCloud functions under the covers. This can also be usefull if a problem arises, to debug the issue or at least know where to start.

Nextcloud PHP Install

Download and unzip the NextCloud release. Change the release number you find.

Change :

  • VER : NextCloud version

File: ~/nextcloud-install.sh

## https://docs.nextcloud.com/server/latest/admin_manual/installation/command_line_installation.html
##
VER=nextcloud-24.0.7
##---------------------------------------------------
## Download
curl https://download.nextcloud.com/server/releases/${VER}.zip -o ${VER}.zip
##
##---------------------------------------------------
## Extract
DIR=$(pwd)
sudo mkdir -p /var/www/nextcloud
cd /var/www
sudo unzip ${DIR}/${VER}.zip
sudo chown -R www-data:www-data /var/www/nextcloud/

Package Dependency Install

Now install the Debian/Ubuntu packages.

$ sudo apt-get install zip libapache2-mod-php php-gd php-json php-pgsql php-curl php-mbstring php-intl php-imagick php-xml php-zip php-bcmath php-gmp zip php-apcu

Create Empty PostgreSQL Database

Here we create an empty database for NextCloud.

Reference: https://docs.nextcloud.com/server/20/admin_manual/configuration_database/linux_database_configuration.html

PostgreSQL PHP configuration

File: /etc/php7/conf.d/pgsql.ini

# configuration for PHP PostgreSQL module
extension=pdo_pgsql.so
extension=pgsql.so

[PostgresSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
~
:wq

Create nextcloud database

Change :

  • nextclouddb : Your PostgreSQL database owner
$ sudo -u postgres psql -d template1
template1=# 
CREATE USER nextclouddb CREATEDB;
CREATE DATABASE nextcloud OWNER nextclouddb;
\q

PostgreSQL Database Authentication

Recommend using a Database Password, that's what I do. Change :

  • nextclouddb : PostgreSQL database owner
  • ItsABigPaswordToo : password for the database owner

No Database Password

A Nextcloud instance configured with PostgreSQL would contain the path to the socket on which the database is running as the hostname, the system username the PHP process is using, and an empty password to access it, and the name of the database. The config/config.php as created by the Installation wizard would therefore contain entries like this:

File: /var/www/nextcloud/config/config.php

~
  "dbtype"        => "pgsql",
  "dbname"        => "nextcloud",
  "dbuser"        => "nextclouddb",
  "dbpassword"    => "",
  "dbhost"        => "/var/run/postgresql",
  "dbtableprefix" => "oc_",
~
:wq

Note: The host actually points to the socket that is used to connect to the database. Using localhost here will not work if postgreSQL is configured to use peer authentication. Also note that no password is specified, because this authentication method doesn’t use a password.

Database Password

If you use another authentication method (not peer), you’ll need to use the following steps to get the database setup: Now you need to create a database user and the database itself by using the PostgreSQL command line interface. The database tables will be created by Nextcloud when you login for the first time.

$ sudo -u postgres psql -d postgres
postgres=#
ALTER USER nextclouddb WITH PASSWORD 'ItsABigPaswordToo';
drop database nextcloud;
CREATE DATABASE nextcloud TEMPLATE template0 ENCODING 'UNICODE';
ALTER DATABASE nextcloud OWNER TO nextclouddb;
GRANT ALL PRIVILEGES ON DATABASE nextcloud TO nextclouddb;

SHOW hba_file;
              hba_file               
-------------------------------------
 /etc/postgresql/12/main/pg_hba.conf
(1 row)

\q

Set the PostgreSQL Host Based Authentication (hba) for a database user nextclouddb on database nextcloud

File: /etc/postgresql/12/main/pg_hba.conf 

~
# nextcloud
host    nextcloud     nextclouddb    192.168.0.2/32    md5  # or `scram-sha-256` instead of `md5` if you use that
hostssl nextcloud     nextclouddb    192.168.0.2/32    md5  # or `scram-sha-256` instead of `md5` if you use that
~
:wq

A Nextcloud instance configured with PostgreSQL would contain the hostname on which the database is running, a valid username and password to access it, and the name of the database. The config/config.php as created by the Installation wizard would therefore contain entries like this:

File: /var/www/nextcloud/config/config.php

~
  "dbtype"        => "pgsql",
  "dbname"        => "nextcloud",
  "dbuser"        => "nextclouddb",
  "dbpassword"    => "ItsABigPaswordToo",
  "dbhost"        => "localhost",
  "dbtableprefix" => "oc_",
~
:wq

PostgreSQL Database Populate NextCloud Metadata

Next we populate the empty database with NextCloud metadata.

File: ~/nextcloud-db-populate.sh 

#!/bin/bash
#####################################################################################
cd /var/www/nextcloud
PASS="ItsABigPaswordToo"
sudo -u www-data php ./occ  maintenance:install --database \
 "pgsql" --database-name "nextcloud"  --database-user "nextclouddb" --database-pass \
 "${PASS}" --admin-user "bigcloud" --admin-pass "${PASS}"

Apache Web Server NextCloud Configuration

You can change the alias /nextcloud to a more interesting name if you like.

File: /etc/apache2/sites-available/nextcloud.conf

Alias /nextcloud "/var/www/nextcloud/"

<VirtualHost *:80>
  DocumentRoot /var/www/html/
  ServerName  www.example.com
</VirtualHost>

<VirtualHost _default_:443>
  DocumentRoot /var/www/html/
  ServerName  www.example.com

  <Directory /var/www/nextcloud/>
    Require all granted
    AllowOverride All
    Options FollowSymLinks MultiViews

    <IfModule mod_dav.c>
      Dav off
    </IfModule>
  </Directory>

# Use HTTP Strict Transport Security to force client to use secure connections only      
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains;"
SSLEngine on

# certbot
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

Enable nextcloud.conf

$ sudo a2ensite nextcloud

Apache Web Server Locale Settings

Some installs of Apache2 do not enable the locale properly, you can set it like this:

File: /etc/apache2/envvars

~
## Uncomment the following line to use the system default locale instead:
. /etc/default/locale
~

Restart apache to pick up changes

$ sudo systemctl restart apache2

NextCloud Trusted Domains and E-Mail Server Connection

Make sure you have trusted_domains set to the NextCloud host, and also the E-Mail host and settings.

File: /var/www/nextcloud/config/config.php

#  edit the "trusted_domains" setting in /var/www/nextcloud/config/config.php
# ...
  array (
    0 => 'localhost',
    1 => '192.168.0.5',
    2 => 'www.example.com",
  ),
~
  'default_phone_region' => 'US',
  'mail_from_address' => 'nextcloud',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => 'mail.example.com',
  'mail_smtphost' => '192.168.0.25',
  'mail_smtpport' => '25',
~
:wq

NextCloud PHP Configuration

Enable PHP large memory.

File: /etc/php/7.4/apache2/php.ini

~
;memory_limit = 128M
memory_limit = 512M
~
:wq

NextCloud Redis Caching Configuration

Enable Redis caching.

Reference: https://docs.nextcloud.com/server/22/admin_manual/configuration_server/caching_configuration.html

Install Redis

$ sudo apt-get install redis-server php-redis

Uncomment the Unix socket server.sock in the redis.conf file

File: /etc/redis/redis.conf

~
unixsocket /var/run/redis/redis-server.sock
~
:wq

Allow www-data access to the socket

$ sudo usermod -a -G redis www-data

Add memcache entries in NextCloud config

File: /var/www/nextcloud/config/config.php

~
  'memcache.locking' => '\OC\Memcache\Redis',
  'memcache.local' => '\OC\Memcache\APCu',
  'memcache.distributed' => '\OC\Memcache\Redis',
  'redis' => [
     'host'     => '/run/redis/redis-server.sock',
     'port'     => 0,
     'timeout'  => 1.5,
  ],
~
:wq

Set Redis session handling for Nextcloud

File: /etc/php/7.4/apache2/php.ini

~
redis.session.locking_enabled=1
redis.session.lock_retries=-1
redis.session.lock_wait_time=10000
~
:wq

Move the shared files in NextCloud to NAS

These are the files that sync from your machine to NextCloud, like pictures, documents, music, etc. You we be able to see the files in a normal filesystem, just refrain from making any changes.

Log into PostgreSQL and connect to the nextcloud database.

$ sudo -u postgres psql -d nextcloud

nextcloud=# select * from oc_storages;
 numeric_id |               id                | available | last_checked 
------------+---------------------------------+-----------+--------------
          1 | home::bigcloud                  |         1 |             
          2 | local::/var/www/nextcloud/data/ |         1 |             
(3 rows)

The local::/var/www/nextcloud/data id is the default location for sync data files.

  1. Stop apache and redis
$ sudo systemctl stop apache2
$ sudo systemctl stop redis
  1. Make directory on nas. Log into the nas as root because you need to change permissions of the file directory. Assuming the nas mount point is /mnt/vol01/nfs_share
$ sudo mkdir -p /mnt/vol01/nfs_share/nextcloud
$ sudo chown -R www-data:www-data /mnt/vol01/nfs_share/nextcloud
  1. Copy the data on the NextCloud server to the nfs mount. This assumes the nfs mount point is /data.
$ sudo -u www-data rsync -rav /var/www/nextcloud/data/ /data/nextcloud/
  1. Ensure it is owned by www-data:www-data
$ sudo -u www-data ls -lrt /data/nextcloud/
total 51
-rw-rw-r--  1 www-data www-data      0 Sep 22 12:16 index.html
drwxr-xr-x  4 www-data www-data      4 Sep 22 12:36 bigcloud
drwxr-xr-x  3 www-data www-data      3 Sep 22 16:59 __groupfolders
drwxr-xr-x 11 www-data www-data     11 Sep 22 18:16 appdata_oc3hnmjliksp
-rw-r-----  1 www-data www-data 290001 Sep 23 12:26 nextcloud.log
  1. Rename old data
$ sudo mv /var/www/nextcloud/data /var/www/nextcloud/data.old
  1. Update NextCloud config for the new directory

File: /var/www/nextcloud/config/config.php

~
##     from
##     'datadirectory' => '/var/www/nextcloud/data',
##     to
      'datadirectory' => '/data/nextcloud',
~
  1. Update database:
$ sudo -u postgres -d nextcloud
 
nextcloud=# select * from oc_storages;
 numeric_id |               id                | available | last_checked 
------------+---------------------------------+-----------+--------------
          1 | home::bigcloud                |         1 |             
          2 | local::/var/www/nextcloud/data/ |         1 |             
(3 rows)

nextcloud=# update oc_storages set id='local::/data/nextcloud/' where id='local::/var/www/nextcloud/data/';
UPDATE 1
nextcloud=# select * from oc_storages;
 numeric_id |           id            | available | last_checked 
------------+-------------------------+-----------+--------------
          1 | home::bigcloud        |         1 |             
          2 | local::/data/nextcloud/ |         1 |             
(3 rows)
  1. Start redis and apache
$ sudo systemctl start redis
$ sudo systemctl start apache2

Nextcloud Command Line Client

This CLI tool allows syncing files from the local machine to NextCloud.

Install

$ sudo apt-get install nextcloud-desktop-cmd

Usage

 nextcloudcmd --help
nextcloudcmd - command line Nextcloud client tool

Usage: nextcloudcmd [OPTION] <source_dir> <server_url>

A proxy can either be set manually using --httpproxy.
Otherwise, the setting from a configured sync client will be used.

Options:
  --silent, -s           Don't be so verbose
  --httpproxy [proxy]    Specify a http proxy to use.
                         Proxy is http://server:port
  --trust                Trust the SSL certification.
  --exclude [file]       Exclude list file
  --unsyncedfolders [file]    File containing the list of unsynced remote folders (selective sync)
  --user, -u [name]      Use [name] as the login name
  --password, -p [pass]  Use [pass] as password
  -n                     Use netrc (5) for login
  --non-interactive      Do not block execution with interaction
  --nonshib              Use Non Shibboleth WebDAV authentication
  --davpath [path]       Custom themed dav path, overrides --nonshib
  --max-sync-retries [n] Retries maximum n times (default to 3)
  --uplimit [n]          Limit the upload speed of files to n KB/s
  --downlimit [n]        Limit the download speed of files to n KB/s
  -h                     Sync hidden files, do not ignore them
  --version, -v          Display version and exit
  --logdebug             More verbose logging

To synchronize the Nextcloud directory Music to the local directory media/music, through a proxy listening on port 8080, and on a gateway machine using IP address 192.168.178.1, the command line would be:

$ nextcloudcmd --httpproxy http://192.168.178.1:8080 \
              $HOME/media/music \
              https://server/nextcloud/remote.php/webdav/Music

nextcloudcmd will prompt for the user name and password, unless they have been specified on the command line or -n has been passed.

NOTE: --exclude option does not work! (Oct-2022) Need to create/edit a file named .sync-exclude.lst in the top level directory of the client side. Example: 

$ cat cloud/.sync-exclude.lst 
/Pictures/*
Pictures/*

Debug

  • Problem: External storage option for SMB/CIFS.

  • Solution: Enable APP 'External Storage" and install package

    $ sudo apt-get install smbclient

  • Problem: Backup calendar and contacts

  • Solution: Use vdirsync (see below) Offline Copy of Contacts and Calendar

  • Problem: Contacts and Calendar access from iOS

  • Solution: Create or modify htaccess file in web root directory to perform http redirect (301)

    File: /var/www/html/.htaccess 

    <IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteRule ^\.well-known/carddav   /remote.php/dav [R=301,L]
  RewriteRule ^\.well-known/caldav    /remote.php/dav [R=301,L]
  RewriteRule ^\.well-known/webfinger /index.php/.well-known/webfinger [R=301,L]
  RewriteRule ^\.well-known/nodeinfo  /index.php/.well-known/nodeinfo [R=301,L]
</IfModule>

Talk from Command Line

This is a way to send messages to the phone from Linux command line. Requires the 'NextCloud Talk' iOS app from the Apple App store and 'Mattermost' app from the NextCloud App store.

Talk Mattermost Shell Script

After Nextcloud 27, Talk Mattermost app does not work, and is not needed to send messages through the command line.

File: ~/talk_mattermost.sh

#!/bin/bash
#----------------------------------------------------
# File: talk_mattermost.sh
#
# Usage: talk_mattermost.sh <msg>
#
# Purpose: Send message to nextcloud talk client
#
# Dependencies: 
# Version Nextcloud 26 or below:
#  1) In NextCloud apps, search for Mattermost
# 
#  2) Click install it
# 
#  3) In Settings, enable it
#
# All Versions: 
# 4) Create a dedicated user in Nextcloud; i.e.: robot
#     with password under “User/Security”.
#     (a new account specifically for the bot) first.
#     It will not relay messages from yourself if you use your account
# 
# 5) Create a Nextloud Talk channel
#     - new conversasion (e.g.: robotic) while logged in as new user 'robot'
#     - under ... [options] enable MatterMost -> Nextcloud Talk)
#       with the user for the automatic service
#     - add users to whom the push messages should be sent.
#       (you will see automatic user 'bridge-bot' as a participant)
# 
# 6) Open the channel and *copy the channel id* from the URL
#     (https://<address_to_nextcloud_service>/index.php/call/<channel_id>).
# 
# 7) And now follows the magic PHP-code part, which has to be copied somewhere on the server.
# 
# <?php
# 	function NextcloudTalk_SendMessage($channel_id, $message) {
# 		$SERVER = "https://<address_to_nextcloud_service>";
# 		$USER = "<nextcloud_robotic_user>";
# 		$PASS = "<application_password>";
# 
# 		// notify hack
# 		$data = array(
# 			"token" => $channel_id,
# 			"message" => $message,
# 			"actorDisplayName" => "PYTHON-NOTIFICATION",
# 			"actorType" => "",
# 			"actorId" => "",
# 			"timestamp" => 0,
# 			"messageParameters" => array()
# 		);
# 
# 		$payload = json_encode($data);
# 
# 		$ch = curl_init($SERVER . '/ocs/v2.php/apps/spreed/api/v1/chat/' . $channel_id);
# 
# 		curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
# 		curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
# 		curl_setopt($ch, CURLINFO_HEADER_OUT, true);
# 		curl_setopt($ch, CURLOPT_POST, true);
# 		curl_setopt($ch, CURLOPT_POSTFIELDS, $payload);
# 		curl_setopt($ch, CURLOPT_USERPWD, "$USER:$PASS");
# 		curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
# 
# 		// Set HTTP Header
# 		curl_setopt($ch, CURLOPT_HTTPHEADER, array(
# 			'Content-Type: application/json',
# 			'Content-Length: ' . strlen($payload),
# 			'Accept: application/json',
# 			'OCS-APIRequest: true')
# 		);
# 
# 		$result = curl_exec($ch);
# 		curl_close($ch);
# 
# 	}
# 
# 	$token = $argv[1];
# 	$message = $argv[2];
# 
# 	NextcloudTalk_SendMessage($token, $message);
# ?>
# 
# 8) Test service from command line
# php <path_to_file>/nextcloudmessage.php <channel_id> <message>
# 
# Reference: https://www.developercookies.net/push-notification-service-with-nextcloud-talk/
#            https://github.com/42wim/matterbridge#configuration
#
# Date     Author     Description
# ----     ------     -----------
# Dec-2021 Don Cohoon Created
# Jun-2023 Don Cohoon Talk Mattermost app not available/needed on Nextcloud v27
#----------------------------------------------------
CHANNEL_ID=***********
/usr/bin/php /data/talk_mattermost.php ${CHANNEL_ID} "${@}"

Talk Mattermost PHP Script

<?php
	function NextcloudTalk_SendMessage($channel_id, $message) {
		$SERVER = "https://www.example.com/";
		$USER = "robot";
		$PASS = "*************";

		// notify hack
		$data = array(
			"token" => $channel_id,
			"message" => $message,
			"actorDisplayName" => "PYTHON-NOTIFICATION",
			"actorType" => "",
			"actorId" => "",
			"timestamp" => 0,
			"messageParameters" => array()
		);

		$payload = json_encode($data);

		$ch = curl_init($SERVER . '/ocs/v2.php/apps/spreed/api/v1/chat/' . $channel_id);

		curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
		curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
		curl_setopt($ch, CURLINFO_HEADER_OUT, true);
		curl_setopt($ch, CURLOPT_POST, true);
		curl_setopt($ch, CURLOPT_POSTFIELDS, $payload);
		curl_setopt($ch, CURLOPT_USERPWD, "$USER:$PASS");
		curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);

		// Set HTTP Header
		curl_setopt($ch, CURLOPT_HTTPHEADER, array(
			'Content-Type: application/json',
			'Content-Length: ' . strlen($payload),
			'Accept: application/json',
			'OCS-APIRequest: true')
		);

		$result = curl_exec($ch);
		curl_close($ch);

	}

	$token = $argv[1];
	$message = $argv[2];

	NextcloudTalk_SendMessage($token, $message);
?>

Federated Sharing Between Nextcloud Servers

If you have a friend using NextCloud and want to share data between you and them, or you have an organization with several instances of NextCloud, it is possible to let them sync between each other. This is called Federation.

The main requirement is that both hosts must have https (SSL/TLS) enabled on their web servers with valid certificates.

In both Nextcloud hosts

  1. Enable the Federation App in NextCloud app admin screen.

  2. Update NextCloud configs, trusted_domains and add the allow_local_remote_servers if both hosts are on the same domain.

File: /var/www/nextcloud/config.php:

'trusted_domains' =>  
  array (  
    0 => '[localhost](http://localhost)',  
    1 => '192.168.0.5',  
    2 => '[one.example.com](http://one.example.com)',  
    3 => '[two.example.com](http://two.example.com)',  
    4 => '[www.example.com](http://www.example.com)',  
    5 => '[example.com](http://example.com)',  
  ),  
\~

'allow_local_remote_servers'=>true,

\~
  1. In NextCloud setting screen: Set global sharing checkboxes to send and receive remote shares

In Sending Nextcloud

In NextCloud file screen: Share file/folder using

<login>@<server>/<URI>

For example, from host www to share with host one user cloud enter this as the shared with name on the File screen:

cloud@one.example.com

In Receiving Nextcloud

Check NextCloud on receiving host, should see share alert pop up, then you NEED to press the ACCEPT button .

Reference:

Offline Copy of Contacts and Calendar

This is a solution for backing up Contacts and Calendars from the NextCloud database to a flat file. The file can then be used for restore back to the database or transfer to another NextCloud instance. Additionally it can be used to access Contacts and Calendars from Linux command line.

Install vdirsyncer package

$ sudo apt-get install vdirsyncer

Configure vdirsyncer

File: /data/vcard/vdirsyncer-nextcloud.conf

[general]
status_path = "~/.vdirsyncer/status/"

[pair contacts_nextcloud_to_local]
a = "my_contacts_local"
b = "my_contacts_nextcloud"
collections = ["from a", "from b"]

[storage my_contacts_local]
type = "filesystem"
path = "~/.contacts/"
fileext = ".vcf"

[storage my_contacts_nextcloud]
type = "carddav"

url = "https://www.example.com/"
username = "cloud"
password = "*************"

[pair cal_nextcloud_to_local]
a = "my_cal_local"
b = "my_cal_nextcloud"
collections = ["from a", "from b"]

[storage my_cal_local]
type = "filesystem"
path = "~/.calendars/"
fileext = ".ics"

[storage my_cal_nextcloud]
type = "caldav"

url = "https://www.example.com/"
username = "cloud"
password = "**********"

Run discovery to populate the sync directories and configuration in your $HOME directory

$ vdirsyncer -c /data/vcard/vdirsyncer-nextcloud.conf discover

Create a script to run it. Make sure you created the .htaccess file above.

File:/data/vcard/vdirsyncer.sh

#!/bin/bash
# sudo apt-get install vdirsyncer
#
# One-time only
#vdirsyncer -c vdirsyncer-nextcloud.conf discover
#
# NOTE: Need to add .htaccess to /var/www/html
#  Ref: https://docs.nextcloud.com/server/23/admin_manual/issues/general_troubleshooting.html#service-discovery
#
DIR=/data/vcard
vdirsyncer -c ${DIR}/vdirsyncer-nextcloud.conf sync 2>&1 | mail -s vdirsync mail@example.com

Schedule vdirsyncer

Ensure it is executable:

$ chmod 755 /data/vcard/vdirsyncer.sh

Schedule vdirsync via /etc/cron.daily to backup contacts and calendars.

File: /etc/cron.daily/vdirsyncer

#!/bin/bash
sudo -u bob /data/vcard/vdirsyncer.sh

Contacts from the Command Line

khard command line will search and display your NextCloud contacts using vdirsync files locally.

Reference: https://khard.readthedocs.io/en/latest/

Install

$ sudo apt-get install khard

Configure

First create a configuration file.

File: /data/vcard/khard.conf


# example configuration file for khard version > 0.14.0
# place it under ~/.config/khard/khard.conf
# This file is parsed by the configobj library.  The syntax is described at
# https://configobj.readthedocs.io/en/latest/configobj.html#the-config-file-format

[addressbooks]
[[family]]
path = ~/.contacts/family/
[[friends]]
path = ~/.contacts/friends/

[general]
debug = no
default_action = list
# These are either strings or comma seperated lists
editor = vim, -i, NONE
merge_editor = vimdiff

[contact table]
# display names by first or last name: first_name / last_name / formatted_name
display = first_name
# group by address book: yes / no
group_by_addressbook = no
# reverse table ordering: yes / no
reverse = no
# append nicknames to name column: yes / no
show_nicknames = no
# show uid table column: yes / no
show_uids = yes
# sort by first or last name: first_name / last_name / formatted_name
sort = last_name
# localize dates: yes / no
localize_dates = yes
# set a comma separated list of preferred phone number types in descending priority
# or nothing for non-filtered alphabetical order
preferred_phone_number_type = pref, cell, home
# set a comma separated list of preferred email address types in descending priority
# or nothing for non-filtered alphabetical order
preferred_email_address_type = pref, work, home

[vcard]
# extend contacts with your own private objects
# these objects are stored with a leading "X-" before the object name in the vcard files
# every object label may only contain letters, digits and the - character
# example:
#   private_objects = Jabber, Skype, Twitter
# default: ,  (the empty list)
private_objects = Jabber, Skype, Twitter
# preferred vcard version: 3.0 / 4.0
preferred_version = 3.0
# Look into source vcf files to speed up search queries: yes / no
search_in_source_files = no
# skip unparsable vcard files: yes / no
skip_unparsable = no

Next create a script to run it, pointing to the configuration file and the 'show' verb.

File: ~/khard.sh

#!/bin/bash
# sudo apt-get install khard
#
# Copy khard.conf to ~/.config/khard/khard.conf 
#
# show : allows selection for details
# list : just shows listing then exit
#
khard -c khard.conf show ${1}

echo ""

# Detailed
#khard -c khard.conf show ${1} --format yaml

# dump and exit
#khard -c khard.conf list ${1}

# use contact.yaml as template
#khard -c khard.conf new -i contact.yaml
#khard -c khard.conf edit -i contact.yaml

Run it

Now see it in action.

Sample run, searching for string match

$ ./khard.sh picard
Select contact for Show action
Address book: All
Index    Name                     Phone                                E-Mail    UID   
1        Dr. Picard, Jeffery    HOME,VOICE: 999-555-1212                       57    
2        Picard                 WORK, VOICE, pref: (999) 555-1212              7A    
Enter Index (q to quit): q
Canceled

Calendars from the Command Line

khal is a command line display of NextCloud calendar entries using vdirsync files locally.

Reference: https://khal.readthedocs.io/en/latest/

Install

$ sudo apt-get install khal

Configure

First crete a configuration file that matches your NextCloud calendar names. This example has 2 calendars, personal and bills.

File: /data/vcard/khal.conf

[calendars]

  [[home]]
    path = ~/.calendars/personal/
    color = dark cyan
    priority = 20

  [[bills]]
    path = ~/.calendars/bills/
    color = dark red
    readonly = True

[locale]
local_timezone = America/New_York
default_timezone = America/New_York

# If you use certain characters (e.g. commas) in these formats you may need to
# enclose them in "" to ensure that they are loaded as strings.
timeformat = %H:%M
dateformat = %d-%b-
longdateformat = %d-%b-%Y
datetimeformat =  %d-%b- %H:%M
longdatetimeformat = %d-%b-%Y %H:%M

firstweekday = 0
#monthdisplay = firstday

[default]
default_calendar = home
timedelta = 7d # the default timedelta that list uses
highlight_event_days = True  # the default is False

Next create a script to run it, pointing to the configuration file.

File: ~/khal.sh

#!/bin/bash
khal -c /data/vcard/khal.conf calendar

Run it

Now run the command line calendar. It will use the latest vsyndir files locally.

$ ./khal.sh 
    Mo Tu We Th Fr Sa Su     Today, 03-Jan-2023
Jan 26 27 28 29 30 31  1     21:00-22:00 Gas Bill ⟳
     2  3  4  5  6  7  8     Monday, 09-Jan-2023
     9 10 11 12 13 14 15     18:30-19:30 Trash pickup tomorrow  ⟳
    16 17 18 19 20 21 22     
    23 24 25 26 27 28 29     
Feb 30 31  1  2  3  4  5     
     6  7  8  9 10 11 12     
    13 14 15 16 17 18 19     
    20 21 22 23 24 25 26     
Mar 27 28  1  2  3  4  5     
     6  7  8  9 10 11 12     
    13 14 15 16 17 18 19     
    20 21 22 23 24 25 26     
Apr 27 28 29 30 31  1  2

Apache - Block Malicious Hosts

With NextCloud comes Apache web server, and with that comes strangers knocking on your door. I welcome my friends, family, neighbors and people who just want to look at your front garden. I also limit who has access to my back garden and especially indoors.

With that, here is a way to post a bouncer at your gate. Obtain the IP addresses from your logwatch and logcheck scripts as well as the /var/log/apache2/errors.log. They usually show up as some kind of mal-formed URL request with strange directory patterns.

Examples of mal-formed URLs:

/.DS_Store
/.env
/debug/default/view?panel=config
/ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application
/telescope/requests
/s/633323e2431313e2535323e26393/_/;/META-INF/maven/com.atlassian.jira/jira-webapp-dist/pom.properties
/?rest_route=/wp/v2/users/
/server-status
/.git/config
/.vscode/sftp.json
/info.php
/login.action
/config.json
/v2/_catalog
/api/search?folderIds=0
/about

Script

The script from nitefood at github provides Autonomous System Numbers (ASN) which is a better indicator of all an organizations IP number ranges then the standard whois <ip>, so the firewall.sh script will utilize them as CIDR range blocks to UFW.

In order to use the IPQualityScore API for in-depth threat reporting, it's necessary to sign up for their service (it's free) and get an API token (it will be emailed to you on sign-up), which will entitle you to 5000 free lookups per month.

Reference:

This script will block the IP address range of an organization. Typically when one IP is hacking into systems, others within that domains range will be at it too.

File: ~/linux/firewall.sh

#!/bin/bash
####################################################################
#
# File: firewall.sh
#
# Purpose: get CIDR of IP and block using ufw
#
# Dependencies:
#  sudo apt-get install curl whois bind9-host mtr-tiny jq ipcalc grepcidr nmap ncat aha 
#
#  git clone https://github.com/nitefood/asn 
#
#  Be sure to get an ~/.asn/iqs_token
#  from https://github.com/nitefood/asn#ip-reputation-api-token
#
####################################################################
DIR=/root
LOG=${DIR}/firewall.log
WHO=/tmp/whois.txt
CIDR=/tmp/whois.cidr
IP="${1}"
#
function run_asn() {
  ${DIR}/asn/asn -n ${IP} > ${WHO}
  /usr/bin/cat ${WHO}
  RANGE=$(/usr/bin/cat ${WHO} | /usr/bin/grep 'NET' | /usr/bin/grep '/' | /usr/bin/awk -Fm '{print $6}' | /usr/bin/cut -d" " -f1)
  echo "CDR: ${RANGE}"
  echo "${RANGE}" > ${CIDR}
}
#
if [ ${1} ]; then
  run_asn
else
  echo "Usage: ${0} <IP Address>"
  exit 1
fi
#
/usr/bin/grep -v deaggregate ${CIDR} > ${CIDR}.block
while read -r IP
  do
    echo "Blocking: ${IP}" | tee -a ${LOG}
    sudo /usr/sbin/ufw prepend deny from ${IP} to any 2>&1 |tee -a $LOG
  done < ${CIDR}.block

Make it executable

$ chmod 755 ~/linux/firewall.sh

Usage

$ ~/linux/firewall.sh 43.129.97.125
                                                                                                                                                           
╭──────────────────────────────╮
│ ASN lookup for 43.129.97.125 │
╰──────────────────────────────╯

 43.129.97.125 ┌PTR -
               ├ASN 132203 (TENCENT-NET-AP-CN Tencent Building, Kejizhongyi Avenue, CN)
               ├ORG 6 COLLYER QUAY
               ├NET 43.129.64.0/18 (ACEVILLEPTELTD-SG)
               ├ABU -
               ├ROA ✓ UNKNOWN (no ROAs found)
               ├TYP  Hosting/DC 
               ├GEO Hong Kong, Central and Western District (HK)
               └REP ✓ NONE  SEEN SCANNING 


CDR: 43.129.64.0/18
Rule inserted
43.129.64.0/18 Blocked

Upgrade Nextcloud

Download the latest nextcloud community server software.

$ curl https://download.nextcloud.com/server/releases/latest.zip -o nextcloud.zip

Upgrade manually

If you upgrade from a previous major version please see critical changes first.

Reference: https://docs.nextcloud.com/server/stable/admin_manual/release_notes/index.html#critical-changes

Always start by making a fresh backup and disabling all 3rd party apps.

  • Back up your existing Nextcloud Server database, data directory, and config.php file. (See Backup, for restore information see Restoring backup)

  • Download and unpack the latest Nextcloud Server release (Archive file) from nextcloud.com/install/ into an empty directory outside of your current installation.

$ unzip nextcloud-[version].zip 
# -or- 
$ tar -xjf nextcloud-[version].tar.bz2

  • Stop your Web server.

  • In case you are running a cron-job for nextcloud’s house-keeping disable it by commenting the entry in the crontab file

# Debian
$ sudo crontab -u www-data -e
# RedHat
$ sudo crontab -u apache -e

(Put a # at the beginning of the corresponding line.)

  • Rename your current Nextcloud directory, for example nextcloud-old.

  • Unpacking the new archive creates a new nextcloud directory populated with your new server files. Move this directory and its contents to the original location of your old server. For example /var/www/, so that once again you have /var/www/nextcloud.

  • Copy the config/config.php file from your old Nextcloud directory to your new Nextcloud directory.

If you keep your data/ directory in your nextcloud/ directory, copy it from your old version of Nextcloud to your new nextcloud/. If you keep it outside of nextcloud/ then you don’t have to do anything with it, because its location is configured in your original config.php, and none of the upgrade steps touch it.

If you are using 3rd party application, it may not always be available in your upgraded/new Nextcloud instance. To check this, compare a list of the apps in the new nextcloud/apps/ folder to a list of the of the apps in your backed-up/old nextcloud/apps/ folder. If you find 3rd party apps in the old folder that needs to be in the new/upgraded instance, simply copy them over and ensure the permissions are set up as shown below.

If you have additional apps folders like for example nextcloud/apps-extras or nextcloud/apps-external, make sure to also transfer/keep these in the upgraded folder.

If you are using 3rd party theme make sure to copy it from your themes/ directory to your new one. It is possible you will have to make some modifications to it after the upgrade.

  • Adjust file ownership and permissions:
$ cd /var/www
# Debian
$ sudo chown -R www-data:www-data nextcloud
# RedHat
$ sudo chown -R apache:apache nextcloud
# Both
$ sudo find nextcloud/ -type d -exec chmod 750 {} \;
$ sudo find nextcloud/ -type f -exec chmod 640 {} \;

  • Restart your Web server.

  • Now launch the upgrade from the command line using occ:

$ cd  /var/www/nextcloud/
# Debian
$ sudo -u www-data php /var/www/nextcloud/occ upgrade
# RedHat
$ sudo -u apache php /var/www/nextcloud/occ upgrade

This MUST be executed from within your nextcloud installation directory

The upgrade operation takes a few minutes to a few hours, depending on the size of your installation. When it is finished you will see a success message, or an error message that will tell where it went wrong.

  • Re-enable the nextcloud cron-job. (See step 4 above.)
# Debian
$ sudo crontab -u www-data -e
# RedHat
$ sudo crontab -u apache -e

(Delete the # at the beginning of the corresponding line in the crontab file.)

  • Login and take a look at the bottom of your Admin page to verify the version number. Check your other settings to make sure they’re correct. Go to the Apps page and review the core apps to make sure the right ones are enabled. Re-enable your third-party apps.

Reference: https://docs.nextcloud.com/server/stable/admin_manual/maintenance/manual_upgrade.html

Continue

Now that you have Cloud, consider some Home Automation, like door and window security, lights and more.

Proceed in the order presented, some things are depending on prior setups.

Book Last Updated: 29-March-2024


Cloud - Linux in the House - https://linux-in-the-house.org Creative Commons License