Friday, June 30, 2017

Periodic services with SMF

Periodic services with SMF

Solaris 11.3 extended SMF to support periodic and scheduled services. It was done mostly to ease IPS packagers because you can't just drop a file in /etc/cron.d and it will magically self-assemble to a cronjob (unlike e.g. /etc/logadm.d).

What's the difference between periodic and scheduled? You want a periodic service for tasks that have to run say every 15 minutes.

Each invocation of a periodic service instance start method occurs at a time relative to the last invocation.

And a scheduled service for tasks that have to run at a specific time.

Each invocation of a scheduled service instance start method occurs at a specific absolute time. Use a scheduled service when the task must run at a certain time, such as during off-peak hours.

The svcbundle command was extended as well to create periodic and scheduled service manifests.

For the periodic example we'll use fetchmail to check for new mails every 15 minutes (== 900s).

$ echo $(( 15 * 60 ))
900
$ svcbundle -o periodic_fetchmail.xml -s service-name=site/periodic/fetchmail \
  -s start-method='/usr/bin/fetchmail -s' -s period=900
# cp periodic_fetchmail.xml /lib/svc/manifest/site/
# svcadm restart svc:/system/manifest-import:default
$ svcs -o NRUN svc:/site/periodic/fetchmail:default
NRUN
22:15:27

We could also use a method_credential (see svc.periodicd(8)) so that the manifest runs as the specified user.

Let's create a scheduled service that runs every night around 00:00 o'clock now.

The crontab entry would look like the following.

0 0 * * * /usr/sbin/audit -n

Let's convert that to SMF with svcbundle.

$ svcbundle -o scheduled_auditrotate.xml -s service-name=site/scheduled/auditrotate \
  -s start-method='/usr/sbin/audit -n' \
  -s hour=0 -s minute=0 \
  -s interval=day
# cp scheduled_auditrotate.xml /lib/svc/manifest/site/
# svcadm restart svc:/system/manifest-import:default
$ svcs -o NRUN svc:/site/scheduled/auditrotate:default
NRUN
 0:00:51

We should edit the manifest and remove the dependency on vc:/milestone/multi-user and instead depend on svc:/system/auditd:default and fill in common_name and description.

For the last example we'll follow Oracle's advice and do a monthly scrub of our disks. See Recommended Storage Pool Practices.

Scrub your ZFS storage pools routinely, such as monthly, if you are using datacenter quality services.

Let's do another scheduled service which runs every month.

$ svcbundle -o scheduled_rpoolscrub.xml -s service-name=site/scheduled/rpoolscrub \
  -s start-method='/usr/sbin/zpool scrub rpool' \
  -s day_of_month=13 -s hour=4 \
  -s interval=month
# cp scheduled_rpoolscrub.xml /lib/svc/manifest/site/
# svcadm restart svc:/system/manifest-import:default

... wait till it executed

$ svcs -o LRUN svc:/site/scheduled/rpoolscrub:default
LRUN
 4:17:49
$ svcs -o NRUN svc:/site/scheduled/rpoolscrub:default
NRUN
Jul_13

Also note there are log files in /var/svc/log in case something goes wrong.

$ svcs -Lv site/scheduled/rpoolscrub
svc:/site/scheduled/rpoolscrub:default (Monthly rpool scrub)
Logfile not found for svc:/site/scheduled/rpoolscrub:default. Error: unknown error.

$ tail /var/svc/log/site-scheduled-rpoolscrub:default
Jun 13 04:17:49/6:Executing start method ("/usr/sbin/zpool scrub rpool")
Jun 13 04:17:55/5:Method "start" exited with status 0.

When you're an IPS packager, don't forget to mogrify your SMF manifests.

<transform dir path=lib/svc/manifest -> set group sys>
<transform file path=lib/svc/manifest -> set group sys>
<transform file path=lib/svc/method/.* -> set mode 0555>
<transform file path=(var|lib)/svc/manifest/.*\.xml$ -> \
    default restart_fmri svc:/system/manifest-import:default>

Links

Tuesday, June 13, 2017

How to build software on Solaris 11/SPARC

How to build software on Solaris 11/SPARC

Today we'll figure out the default build flags used by the Solaris Userland Consolidation.

We already used them while building nginx, see nginx on Solaris 11.3 SRU 19 with ECC crypto and HTTP/2 support.

We need a compiler before we can build anything. My preferred choice is the Solaris Developer Studio compiler. You can download a tarball at Developer Studio or request a solarisstudio repository certificate at Oracle Repositories.

Either way, you should have a decent compiler now.

$ /opt/developerstudio12.5/bin/cc -V
cc: Studio 12.5 Sun C 5.14 SunOS_sparc 2016/05/31

Good. Let get the default 64bit *FLAGS used to build software from shared-macros.mk.

# Enables large file support for components that have no other means of doing
# so.  Use CPP_LARGEFILES and not the .32/.64 variety directly
CPP_LARGEFILES.64 := $(shell getconf LFS64_CFLAGS)
CPP_LARGEFILES =  $(CPP_LARGEFILES.$(BITS))

# XPG6 mode.  This option enables XPG6 conformance, plus extensions.
# Amongst other things, this option will cause system calls like
# popen (3C) and system (3C) to invoke the standards-conforming
# shell, /usr/xpg4/bin/sh, instead of /usr/bin/sh.  Add studio_XPG6MODE to
# CFLAGS instead of using this directly
CPP_XPG6MODE= -D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 -D_XPG6

This should equal the following CPPFLAGS:

CPPFLAGS="$(getconf LFS64_CFLAGS) -D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 -D_XPG6"

Let's do the same for CFLAGS.

BITS ?=   64

MACH :=  $(shell uname -p)
MACH64_1 = $(MACH:sparc=sparcv9)

CC_BITS = -m$(BITS)

studio_XBITS.sparc.64 += -xarch=sparcvis -xchip=ultra2
studio_XBITS = $(studio_XBITS.$(MACH).$(BITS))

# Turn on recognition of supported C99 language features and enable the 1999 C
# standard library semantics of routines that appear in both the 1990 and
# 1999 C standard. To use set studio_C99MODE=$(studio_C99_ENABLE) in your
# component Makefile.
studio_C99_ENABLE =  -xc99=all

# Allow zero-sized struct/union declarations and void functions with return
# statements.
studio_FEATURES_EXTENSIONS = -features=extensions

studio_OPT.sparc.64 ?= -xO4
studio_OPT ?=  $(studio_OPT.$(MACH).$(BITS))

# Studio PIC code generation.  Use CC_PIC instead to select PIC code generation.
studio_PIC =  -KPIC -DPIC

# The Sun Studio 11 compiler has changed the behaviour of integer
# wrap arounds and so a flag is needed to use the legacy behaviour
# (without this flag panics/hangs could be exposed within the source).
# This is used through studio_IROPTS, not the 'sparc' variety.
studio_IROPTS.sparc = -W2,-xwrap_int
studio_IROPTS =  $(studio_IROPTS.$(MACH))

# Control register usage for generated code.  SPARC ABI requires system
# libraries not to use application registers.  x86 requires 'no%frameptr' at
# x04 or higher.

# We should just use -xregs but we need to workaround 7030022. Note
# that we can't use the (documented) -Wc,-xregs workaround because
# libtool really hates -Wc and thinks it should be -Wl. Instead
# we use an (undocumented) option which actually happens to be what
# CC would use.
studio_XREGS.sparc = -Qoption cg -xregs=no%appl
studio_XREGS =  $(studio_XREGS.$(MACH))

# Set data alignment on sparc to reasonable values, 8 byte alignment for 32 bit
# objects and 16 byte alignment for 64 bit objects.  This is added to CFLAGS by
# default.
studio_ALIGN.sparc.64 = -xmemalign=16s
studio_ALIGN =  $(studio_ALIGN.$(MACH).$(BITS))

# Studio shorthand for building multi-threaded code,  enables -D_REENTRANT and
# linking with threadin support.  This is added to CFLAGS by default, override
# studio_MT to turn this off.
studio_MT =  -mt

CFLAGS.studio += $(studio_OPT) $(studio_XBITS) $(studio_XREGS) \
   $(studio_IROPTS) $(studio_C99MODE) $(studio_ALIGN) \
   $(studio_MT)

This should equal the following CFLAGS:

CFLAGS="-m64 -xO4 -xarch=sparcvis -xchip=ultra2 \
  -Qoption cg -xregs=no%appl -W2,-xwrap_int -xc99=all -xmemalign=16s \
  -mt -KPIC -DPIC"

And last but not least, for LDFLAGS.

# set the bittedness that we want to link
LD_BITS = -$(BITS)

# eliminate unreferenced dynamic dependencies
LD_Z_IGNORE =  -zignore

# eliminate comments
LD_Z_STRIP_CLASS = -zstrip-class=comment

# use direct binding
LD_B_DIRECT =  -Bdirect

# build a PIE binary
# to enable creating a PIE binary, add LD_Z_PIE_MODE = $(LD_Z_PIE_ENABLE)
# to the component makefile, and ensure that it's built PIC (CC_PIC_ENABLE).
LD_Z_PIE_ENABLE = -ztype=pie

# by default, turn on Address Space Layout Randomization, non-executable
# stack and non-executable heap for ELF executables;
ASLR_ENABLE =    -zaslr=enable
NXSTACK_ENABLE =  -znxstack=enable
NXHEAP_ENABLE =   -znxheap=enable

# Create a non-executable bss segment when linking.
LD_MAP_NOEXBSS.sparc = -M /usr/lib/ld/map.noexbss

# Create a non-executable data segment when linking.  Due to PLT needs, the
# data segment must be executable on sparc, but the bss does not.
# see mapfile comments for more information
LD_MAP_NOEXDATA.sparc = $(LD_MAP_NOEXBSS.$(MACH))

# Page alignment
LD_MAP_PAGEALIGN = -M /usr/lib/ld/map.pagealign

# Default linker options that everyone should get.  Do not add additional
# libraries to this macro, as it will apply to everything linked during the
# component build.
LD_OPTIONS += $(LD_MAP_NOEXDATA.$(MACH)) \
  $(LD_MAP_PAGEALIGN) $(LD_B_DIRECT) $(LD_Z_IGNORE) \
  $(LD_Z_STRIP_CLASS)

This should equal the following LDFLAGS:

LDFLAGS="-m64 -M /usr/lib/ld/map.noexbss \
  -M /usr/lib/ld/map.pagealign -Bdirect -zignore \
  -zstrip-class=comment -ztype=pie \
  -zaslr=enable -znxstack=enable -znxheap=enable"

Some software also doesn't like the Solaris nm, so we'll use GNU nm instead. See configure.mk.

# temporarily work around some issues
CONFIGURE_ENV += "ac_cv_func_realloc_0_nonnull=yes"
CONFIGURE_ENV += "NM=/usr/gnu/bin/nm"

Putting everything together, here is an example of how I usually invoke configure when building FOSS:

$ getconf PATH
/usr/xpg6/bin:/usr/xpg4/bin:/usr/ccs/bin:/usr/bin:/opt/developerstudio12.5/bin:...
$ PATH=$(getconf PATH) ./configure PKG_CONFIG_PATH=/usr/lib/64/pkgconfig NM=/usr/gnu/bin/nm SHELL=bash MAKE=gmake \
  CPPFLAGS="$(getconf LFS64_CFLAGS) -D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 -D_XPG6" \
  CC=cc CFLAGS='-m64 -xO4 -xarch=sparcvis2 -Qoption cg -xregs=no%appl -W2,-xwrap_int \
    -xc99=all -xmemalign=16s -mt -KPIC -DPIC' \
  LDFLAGS='-m64 -M /usr/lib/ld/map.noexbss -M /usr/lib/ld/map.pagealign \
     -zaslr=enable -znxstack=enable -znxheap=enable \
     -Bdirect -zignore -zstrip-class=comment -ztype=pie' \
   --prefix=/opt/local ...
$ PATH=$(getconf PATH) gmake
...

Links

389 Directory Server 1.3.x LDAP client authentication

389 Directory Server 1.3.x LDAP client authentication Last time we did a multi-master replication setup, see 389 Directory Server 1.3.x Repl...