Postgresql
Table of Contents
Postgresql relational SQL database for systems. This is a great database with high reliablity, performance and extendable. External extensions allow even more customizations.
- User Roles
- Stored Procedures
- JSON native support
- Filesystem access
- Federation with other databases
- Physical and logical replication
- Partitioning
- Small system requirements
Reference: https://www.postgresql.org/
Installation (version 13)
The current version is 15, and Ubuntu 22.04 installs 14.
Install
This example is version 13, but it has not changed the package install command.
$ sudo apt-get install postgresql postgresql-server
Enable auto startup
Yea, still example on version 13, but should be the same on 14 and 15.
$ sudo systemctl start postgresql@13-main
$ sudo systemctl status postgresql@13-main
- RedHat
All files will be stuffed into a single directory by default (/var/lib/pgsql/data); data, config, log, etc...
If you want to have multiple versions, or upgrade versions, read the file /usr/share/doc/postgresql/README.rpm-dist [1]. RedHat gives you more control over the placemant and layout of the postgres database(s).
See pg_lscluster.sh [2] below for more info on versions.
$ sudo -u postgres /usr/bin/postgresql-setup --initdb
* Initializing database in '/var/lib/pgsql/data'
* Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log
$ sudo systemctl enable --now postgresql
$ sudo systemctl status postgresql
Check out all these Options
Check the databases installed, main (postgres) and two templates and list out available command line directives (they start with the backslash (\) character)
$ sudo -u postgres psql
postgres-# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------+---------+-----------------------
postgres | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
# \?
General
\copyright show PostgreSQL usage and distribution terms
\crosstabview [COLUMNS] execute query and display results in crosstab
\errverbose show most recent error message at maximum verbosity
\g [(OPTIONS)] [FILE] execute query (and send results to file or |pipe);
\g with no arguments is equivalent to a semicolon
\gdesc describe result of query, without executing it
\gexec execute query, then execute each value in its result
\gset [PREFIX] execute query and store results in psql variables
\gx [(OPTIONS)] [FILE] as \g, but forces expanded output mode
\q quit psql
\watch [SEC] execute query every SEC seconds
Help
\? [commands] show help on backslash commands
\? options show help on psql command-line options
\? variables show help on special variables
\h [NAME] help on syntax of SQL commands, * for all commands
Query Buffer
\e [FILE] [LINE] edit the query buffer (or file) with external editor
\ef [FUNCNAME [LINE]] edit function definition with external editor
\ev [VIEWNAME [LINE]] edit view definition with external editor
\p show the contents of the query buffer
\r reset (clear) the query buffer
\s [FILE] display history or save it to file
\w FILE write query buffer to file
Input/Output
\copy ... perform SQL COPY with data stream to the client host
\echo [-n] [STRING] write string to standard output (-n for no newline)
\i FILE execute commands from file
\ir FILE as \i, but relative to location of current script
\o [FILE] send all query results to file or |pipe
\qecho [-n] [STRING] write string to \o output stream (-n for no newline)
\warn [-n] [STRING] write string to standard error (-n for no newline)
Conditional
\if EXPR begin conditional block
\elif EXPR alternative within current conditional block
\else final alternative within current conditional block
\endif end conditional block
Informational
(options: S = show system objects, + = additional detail)
\d[S+] list tables, views, and sequences
\d[S+] NAME describe table, view, sequence, or index
\da[S] [PATTERN] list aggregates
\dA[+] [PATTERN] list access methods
\dAc[+] [AMPTRN [TYPEPTRN]] list operator classes
\dAf[+] [AMPTRN [TYPEPTRN]] list operator families
\dAo[+] [AMPTRN [OPFPTRN]] list operators of operator families
\dAp[+] [AMPTRN [OPFPTRN]] list support functions of operator families
\db[+] [PATTERN] list tablespaces
\dc[S+] [PATTERN] list conversions
\dC[+] [PATTERN] list casts
\dd[S] [PATTERN] show object descriptions not displayed elsewhere
\dD[S+] [PATTERN] list domains
\ddp [PATTERN] list default privileges
\dE[S+] [PATTERN] list foreign tables
\des[+] [PATTERN] list foreign servers
\det[+] [PATTERN] list foreign tables
\deu[+] [PATTERN] list user mappings
\dew[+] [PATTERN] list foreign-data wrappers
\df[anptw][S+] [FUNCPTRN [TYPEPTRN ...]]
list [only agg/normal/procedure/trigger/window] functions
\dF[+] [PATTERN] list text search configurations
\dFd[+] [PATTERN] list text search dictionaries
\dFp[+] [PATTERN] list text search parsers
\dFt[+] [PATTERN] list text search templates
\dg[S+] [PATTERN] list roles
\di[S+] [PATTERN] list indexes
\dl list large objects, same as \lo_list
\dL[S+] [PATTERN] list procedural languages
\dm[S+] [PATTERN] list materialized views
\dn[S+] [PATTERN] list schemas
\do[S+] [OPPTRN [TYPEPTRN [TYPEPTRN]]]
list operators
\dO[S+] [PATTERN] list collations
\dp [PATTERN] list table, view, and sequence access privileges
\dP[itn+] [PATTERN] list [only index/table] partitioned relations [n=nested]
\drds [ROLEPTRN [DBPTRN]] list per-database role settings
\dRp[+] [PATTERN] list replication publications
\dRs[+] [PATTERN] list replication subscriptions
\ds[S+] [PATTERN] list sequences
\dt[S+] [PATTERN] list tables
\dT[S+] [PATTERN] list data types
\du[S+] [PATTERN] list roles
\dv[S+] [PATTERN] list views
\dx[+] [PATTERN] list extensions
\dX [PATTERN] list extended statistics
\dy[+] [PATTERN] list event triggers
\l[+] [PATTERN] list databases
\sf[+] FUNCNAME show a function's definition
\sv[+] VIEWNAME show a view's definition
\z [PATTERN] same as \dp
Formatting
\a toggle between unaligned and aligned output mode
\C [STRING] set table title, or unset if none
\f [STRING] show or set field separator for unaligned query output
\H toggle HTML output mode (currently off)
\pset [NAME [VALUE]] set table output option
(border|columns|csv_fieldsep|expanded|fieldsep|
fieldsep_zero|footer|format|linestyle|null|
numericlocale|pager|pager_min_lines|recordsep|
recordsep_zero|tableattr|title|tuples_only|
unicode_border_linestyle|unicode_column_linestyle|
unicode_header_linestyle)
\t [on|off] show only rows (currently off)
\T [STRING] set HTML <table> tag attributes, or unset if none
\x [on|off|auto] toggle expanded output (currently off)
Connection
\c[onnect] {[DBNAME|- USER|- HOST|- PORT|-] | conninfo}
connect to new database (currently "postgres")
\conninfo display information about current connection
\encoding [ENCODING] show or set client encoding
\password [USERNAME] securely change the password for a user
Operating System
\cd [DIR] change the current working directory
\setenv NAME [VALUE] set or unset environment variable
\timing [on|off] toggle timing of commands (currently off)
\! [COMMAND] execute command in shell or start interactive shell
Variables
\prompt [TEXT] NAME prompt user to set internal variable
\set [NAME [VALUE]] set internal variable, or list all if no parameters
\unset NAME unset (delete) internal variable
Large Objects
\lo_export LOBOID FILE
\lo_import FILE [COMMENT]
\lo_list
\lo_unlink LOBOID large object operations
(END)
#\q
Clusters
Before you can do anything, you must initialize a database storage area on disk. This is called a database cluster. A database cluster is a collection of databases that are managed by a single instance of a running database server.
RedHat does not install pg_ctlcluster as it is an extension for Ubuntu package postgresql-common
Clusters are the best way to run several versions of PostgreSQL on the same host,
Reference: https://www.postgresql.org/docs/current/creating-cluster.html
$ pg_ctlcluster
Usage: /usr/bin/pg_ctlcluster <version> <cluster> <action>
$ pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
13 main 5432 down postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log
$ pg_ctlcluster 13 main start
$ pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
13 main 5432 online postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log
- RedHat Version
The location of files is determined by the initdb program, which was run earlier.
File: ~/pg_lscluster.sh
#!/bin/bash
##############################################################
#
# File: pg_lscluster
#
# Purpose: List information on postgresql installed clusters
#
# Reason: pg_lscluster only exists for the postgresql-common
# package on Ubuntu. This script was created for
# non-Ubuntu Linux distributions.
#
# Versions: To do that simply
# 1) Place the postgresql.conf files in versions:
# i.e.: /etc/postgres/<version>/main/postgresql.conf
# 2) Create services with versions:
# i.e.: postgresql@<version>-main.service
# Of course data_directory and log_directory must also have
# versions, and port numbers must be different,
# but this script does not care. ;-)
#
# History
# When Who Description
# ----------- ----------- -----------------------------------------
# 12-Jun-2023 Don Cohoon Created
##############################################################
TMP=$(mktemp)
#-----------
function details() {
SERVICE=${1}
PORT=${2}
#
sudo systemctl status ${SERVICE} > ${TMP}
grep -e active -e PID ${TMP}
# Main PID: 904 (postmaster)
PROCESS=$(grep PID ${TMP}| cut -d'(' -f2| cut -d')' -f1)
# Grab the '-D <directory>' run-time parameter
DIR=$(grep /bin/${PROCESS} ${TMP} | awk '{print $4}')
#
echo
#
sudo -u postgres ls ${DIR}/log >${TMP} 2>/dev/null
if [[ -s ${TMP} ]]; then
echo " Log directory: ${DIR}/log"
elif [[ -d /var/log/postgresql ]]; then
echo " Log directory: /var/log/postgresql"
fi
#
echo
#
sudo -u postgres psql -tq -p ${PORT} <<EOF 2>/dev/null
-- SELECT ' Current Logfile: ' || pg_current_logfile();
SELECT ' Port: ' || setting
FROM pg_settings
WHERE name = 'port';
SELECT ' Data directory: ' || setting
FROM pg_settings
WHERE name = 'data_directory';
SELECT ' Config File: ' || setting
FROM pg_settings
WHERE name = 'config_file';
SELECT ' Host Based Access File: ' || setting
FROM pg_settings
WHERE name = 'hba_file';
EOF
rm ${TMP}
} # details
#-----------
echo
#
if [[ -f /usr/bin/pg_config ]];then
V=$(/usr/bin/pg_config | tail -1 | awk '{print $4}')
else
V=$(/usr/bin/postgres -V | awk '{print $NF}')
fi
echo " Default Version: ${V}"
echo
#
if [[ -d /etc/postgresql/ ]]; then
for VER in $(ls /etc/postgresql/) # return installed versions
do
echo "= = = = => Version: $VER"
PORT=$(grep '^port' /etc/postgresql/${VER}/main/postgresql.conf 2>/dev/null | awk '{print $3}')
details postgresql@${VER}-main.service ${PORT}
done
else # only one default version
details postgresql 5432
fi
#
- Ubuntu
# ~/pg_lscluster.sh
Default Version: 14.8
= = = = => Version: 10
Active: inactive (dead)
Log directory: /var/log/postgresql
= = = = => Version: 12
Active: active (running) since Sat 2023-06-10 16:10:47 EDT; 2 days ago
Main PID: 1485 (postgres)
Log directory: /var/log/postgresql
Port: 5432
Data directory: /var/lib/postgresql/12/main
Config File: /etc/postgresql/12/main/postgresql.conf
Host Based Access File: /etc/postgresql/12/main/pg_hba.conf
= = = = => Version: 14
Active: active (running) since Sat 2023-06-10 16:10:47 EDT; 2 days ago
Main PID: 1462 (postgres)
Log directory: /var/log/postgresql
Port: 5433
Data directory: /var/lib/postgresql/14/main
Config File: /etc/postgresql/14/main/postgresql.conf
Host Based Access File: /etc/postgresql/14/main/pg_hba.conf
- RedHat
# ~/pg_lscluster.sh
Default Version: 13.10
Active: active (running) since Tue 2023-06-13 06:04:30 EDT; 2h 13min ago
Main PID: 904 (postmaster)
Log directory: /var/lib/pgsql/data/log
Port: 5432
Data directory: /var/lib/pgsql/data
Config File: /var/lib/pgsql/data/postgresql.conf
Host Based Access File: /var/lib/pgsql/data/pg_hba.conf
Log File
$ tail -40 /var/log/postgresql/postgresql-13-main.log
2018-11-16 06:04:16.889 EST [1019] LOG: received fast shutdown request
2018-11-16 06:04:16.892 EST [1019] LOG: aborting any active transactions
2018-11-16 06:04:16.897 EST [1019] LOG: worker process: logical replication launcher (PID 1034) exited with exit code 1
2018-11-16 06:04:16.901 EST [1029] LOG: shutting down
2018-11-16 06:04:16.942 EST [1019] LOG: database system is shut down
Configuration File
There are many configuration parameters that affect the behavior of the database system.
Reference: https://www.postgresql.org/docs/current/runtime-config.html
Debian
postgres=# show config_file;
config_file
-----------------------------------------
/etc/postgresql/12/main/postgresql.conf
(1 row)
RedHat
# show config_file;
config_file
-------------------------------------
/var/lib/pgsql/data/postgresql.conf
(1 row)
Host Based Authentication (HBA) File
Client authentication is controlled by a configuration file, which traditionally is named pg_hba.conf and is stored in the database cluster's data directory. (HBA stands for host-based authentication.)
Reference: https://www.postgresql.org/docs/current/auth-pg-hba-conf.html
Debian
# show hba_file;
hba_file
-------------------------------------
/etc/postgresql/12/main/pg_hba.conf
(1 row)
RedHat
# show hba_file;
hba_file
---------------------------------
/var/lib/pgsql/data/pg_hba.conf
(1 row)
psql pager - pspg
The command line paging defaults to less but you can install a much better one.
Pspg supports searching, selecting rows, columns, or block and export selected area to clipboard. Check out the screenshots, installation options and animation on the website. Super useful productivity tool!
Reference: https://github.com/okbob/pspg
Install
$ sudo apt-get install pspg
Helper Script
Create a helper script to run it. Add /usr/local/bin to your PATH if not already.
I defaulted it to a database named
mydb. Change it to your favorite database, or just remove the-d mydbpart of thesudoline.
File: /usr/local/bin/psql.sh
#!/bin/bash
# https://github.com/okbob/pspg
echo "-> Fancy pager (pspg) [-s 1,2,3,4,...20] <- themes"
echo "\set xx '\\\\setenv PAGER \'pspg -bX --no-mouse -s 5\''"
echo ":xx"
echo "-> Default pager (less)"
echo "\set x '\\\\setenv PAGER less'"
echo ":x"
sudo -u postgres psql -d mydb $@
Example
pspg allows paging forward, backward, as well as scrolling right and left with arrow keys.
Execute a file from the script parameter name
$ cat now.sql
select * from now();
$ psql.sh -f now.sql
-> Fancy pager (pspg) [-s 1,2,3,4,...20] <- themes
\set xx '\\setenv PAGER \'pspg -bX --no-mouse -s 5\''
:xx
-> Default pager (less)
\set x '\\setenv PAGER less'
:x
now
-------------------------------
2023-01-28 19:42:07.897614-05
(1 row)
Example using the pager.
Copy & Paste \set xx '\\setenv PAGER \'pspg -bX --no-mouse -s 5\'' into the prompt, hit Enter.
Remember to enter :xx to run the pspg pager.
$ psql.sh
-> Fancy pager (pspg) [-s 1,2,3,4,...20] <- themes
\set xx '\\setenv PAGER \'pspg -bX --no-mouse -s 5\''
:xx
-> Default pager (less)
\set x '\\setenv PAGER less'
:x
psql (14.5 (Ubuntu 14.5-0ubuntu0.22.04.1), server 12.12 (Ubuntu 12.12-0ubuntu0.20.04.1))
Type "help" for help.
mydb=# \set xx '\\setenv PAGER \'pspg -bX --no-mouse -s 5\''
mydb=# :xx
mydb-# select * from mydb.log_file ;
Continue
Now that you have installed a shiny new database, consider how upgrades work.
In our next episode of Linux in the Home.
Proceed in the order presented, some things are depending on prior setups.
Book Last Updated: 29-March-2024
PostgreSQL Database - Linux in the House - https://linux-in-the-house.org
