Skip to content

cronjob: updates required for GRASS 8.5.0 release #1692

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
neteler wants to merge 6 commits into
base: grass8
Choose a base branch
from
+86 −166
Open

cronjob: updates required for GRASS 8.5.0 release #1692

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8dc3962
cronjob: updates required for GRASS 8.5.0 release
neteler May 9, 2026
f1f9a2a
pre-test mkdocs-site.zip artifact before moving on
neteler May 10, 2026
fac221e
add branch name to mkdocs-site.zip and rm file after use
neteler May 10, 2026
b90e6d3
Remove sphinx related steps since done in documentation.yml workflow
neteler May 10, 2026
259a8c9
Merge branch 'grass8' into cronjob_updates_after_g850_release
neteler Jun 14, 2026
22aa2fe
fix shell script permissions
neteler Jun 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 15 additions & 106 deletions utils/cronjobs_osgeo_lxd/cron_grass_current_stable_build_binaries.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,14 @@
# - configures source code and then compiles it
# - packages the binaries
# - generated the install scripts
# - generates the pyGRASS 8 HTML manual
# - generates the user 8 HTML manuals
# - injects DuckDuckGo search field
# - note: the mkdocs based HTML manual pages are generated by GitHub Action
# - copies over generated manual pages to grass-stable/manuals/
# - injects in versioned manual the "canonical" to point to "stable" manual (as seen in the Python manual pages)

# Preparations, on server (neteler@grasslxd:$):
# - git clone the correct GRASS source code version into $HOME/src/
# - install dependencies:
# cd $HOME/src/releasebranch_8_5/ && git pull && sudo apt install $(cat .github/workflows/apt.txt)
# - install further dependencies (not needed for mkdocs used in G85+):
# apt-get install texlive-latex-extra python3-sphinxcontrib.apidoc
# - run this script
# - one time only: cross-link code into web space on grasslxd server:
# cd /var/www/html/
Expand Down Expand Up @@ -159,38 +156,9 @@ echo "GRASS $VERSION compilation done"

######## now GRASS GIS binaries are prepared ####################

#### create module overview (https://trac.osgeo.org/grass/ticket/1203)
#sh utils/module_synopsis.sh

#### generate developer stuff: pygrass docs + gunittest docs
# generate pyGRASS sphinx manual (in docs/html/libpython/)
# including source code
$MYMAKE sphinxdoclib

##
echo "Copy over the manual + pygrass HTML pages:"
mkdir -p $TARGETHTMLDIR
mkdir -p $TARGETHTMLDIR/addons # indeed only relevant the very first compile time
# don't destroy the addons during update
rm -rf /tmp/addons
\mv $TARGETHTMLDIR/addons /tmp
rm -f $TARGETHTMLDIR/*.*
(cd $TARGETHTMLDIR ; rm -rf barscales colortables icons northarrows)
\mv /tmp/addons $TARGETHTMLDIR

cp -rp dist.$ARCH/docs/html/* $TARGETHTMLDIR/
echo "Copied pygrass progman to https://grass.osgeo.org/grass${VERSION}/manuals/libpython/"

# search to be improved with mkdocs or similar; for now we use DuckDuckGo
echo "Injecting DuckDuckGo search field into manual main page..."
(cd $TARGETHTMLDIR/ ; sed -i -e "s+</table>+</table><\!\-\- injected in cron_grass8_relbranch_build_binaries.sh \-\-> <center><iframe src=\"https://duckduckgo.com/search.html?site=grass.osgeo.org%26prefill=Search%20manual%20pages%20at%20DuckDuckGo\" style=\"overflow:hidden;margin:0;padding:0;width:410px;height:40px;\" frameborder=\"0\"></iframe></center>+g" index.html)

# copy important files to web space
cp -p AUTHORS CITING CITATION.cff COPYING GPL.TXT INSTALL.md REQUIREMENTS.md $TARGETDIR/

# clean wxGUI sphinx manual etc
(cd $GRASSBUILDDIR/ ; $MYMAKE cleansphinx)

############
# generate doxygen programmers's G8 manual
cd $GRASSBUILDDIR/
Expand All @@ -199,26 +167,7 @@ $MYMAKE htmldocs-single || (echo "$0 htmldocs-single: an error occurred" ; exit

cd $GRASSBUILDDIR/

#### unused, programmer's manual is only built in "preview" script
## clean old TARGETPROGMAN stuff from last run
#if [ -z "$TARGETPROGMAN" ] ; then
# echo "\$TARGETPROGMAN undefined, error!"
# exit 1
#fi
#mkdir -p $TARGETPROGMAN
#rm -f $TARGETPROGMAN/*.*
#
## copy over doxygen manual
#cp -r html/* $TARGETPROGMAN/
#
#echo "Copied HTML progman to https://grass.osgeo.org/programming${GVERSION}"
## fix permissions
#chgrp -R grass $TARGETPROGMAN/*
#chmod -R a+r,g+w $TARGETPROGMAN/
#chmod -R a+r,g+w $TARGETPROGMAN/*
## bug in doxygen
#(cd $TARGETPROGMAN/ ; ln -s index.html main.html)
#### end unused
#### programmer's manual is only built in "preview" script

##### generate i18N stats for HTML page path:
# note: the gettext POT files are managed in git and OSGeo Weblate
Expand All @@ -238,9 +187,10 @@ cd $TARGETDIR/
cat i18n_stats.txt | grep mod > i18n_stats_mods.txt
cat i18n_stats.txt | grep lib > i18n_stats_libs.txt
cat i18n_stats.txt | grep wxpy > i18n_stats_wxpy.txt
cd $GRASSBUILDDIR

############
# package the GRASS GIS package
cd $GRASSBUILDDIR
$MYMAKE bindist
if [ $? -ne 0 ] ; then
halt_on_error "make bindist."
Expand Down Expand Up @@ -279,8 +229,7 @@ cd $GRASSBUILDDIR
############################################
# compile addons

# update addon repo (addon repo has been cloned twice on the server to
# have separate grass7 and grass8 addon compilation)
# update addon repo
(cd $SOURCE/grass$GMAJOR-addons/; git checkout grass$GMAJOR; git pull origin grass$GMAJOR)
# compile addons
cd $GRASSBUILDDIR
Expand All @@ -291,30 +240,11 @@ sh $MAINDIR/cronjobs/compile_addons_git.sh $GMAJOR \
$MAINDIR/.grass$GMAJOR/addons \
$SOURCE/$BRANCH/bin.$ARCH/grass \
1
mkdir -p $TARGETHTMLDIR/addons/
# copy individual addon html files into one target dir if compiled addon
# has own dir e.g. $MAINDIR/.grass8/addons/db.join/ with bin/ docs/ etc/ scripts/
# subdir
for dir in `find $MAINDIR/.grass$GMAJOR/addons -maxdepth 1 -type d`; do
if [ -d $dir/docs/html ] ; then
if [ "$(ls -A $dir/docs/html/)" ]; then
for f in $dir/docs/html/*; do
# TODO: in cron_grass_preview_build_binaries.sh this is skipped. What is right?
cp $f $TARGETHTMLDIR/addons/
done
fi
fi
done

# TODO: still needed for new mkdocs based manual pages?
sh $MAINDIR/cronjobs/grass-addons-index.sh $GMAJOR $GMINOR $GPATCH $TARGETHTMLDIR/addons/
# copy over hamburger menu assets
cp $TARGETHTMLDIR/grass_logo.png \
$TARGETHTMLDIR/hamburger_menu.svg \
$TARGETHTMLDIR/hamburger_menu_close.svg \
$TARGETHTMLDIR/grassdocs.css \
$TARGETHTMLDIR/addons/
chmod -R a+r,g+w $TARGETHTMLDIR 2> /dev/null

### fetch manual from GH actions
# fetch artifacts for releasebranch_8_X (stable)
bash $HOME/cronjobs/gh_cli_download_artifact.sh $BRANCH
unzip -t /tmp/mkdocs-site_$BRANCH.zip && cd $TARGETHTMLDIR && rm -rf * && unzip -q /tmp/mkdocs-site_$BRANCH.zip && rm -f /tmp/mkdocs-site_$BRANCH.zip

# copy over logs from $MAINDIR/.grass$GMAJOR/addons/logs/
mkdir -p $TARGETMAIN/addons/grass$GMAJOR/logs/
Expand All @@ -324,28 +254,15 @@ cp -p $MAINDIR/.grass$GMAJOR/addons/logs/* $TARGETMAIN/addons/grass$GMAJOR/logs/
$SOURCE/$BRANCH/bin.$ARCH/grass --tmp-project EPSG:4326 --exec $MAINDIR/cronjobs/build-xml.py --build $MAINDIR/.grass$GMAJOR/addons
cp $MAINDIR/.grass$GMAJOR/addons/modules.xml $TARGETMAIN/addons/grass$GMAJOR/modules.xml

# regenerate keywords.html file with addons modules keywords
export ARCH
export ARCH_DISTDIR=$GRASSBUILDDIR/dist.$ARCH
export GISBASE=$ARCH_DISTDIR
export VERSION_NUMBER=$DOTVERSION
python3 $GRASSBUILDDIR/man/build_keywords.py $TARGETMAIN/grass$GMAJOR$GMINOR/manuals/ $TARGETMAIN/grass$GMAJOR$GMINOR/manuals/addons/
unset ARCH ARCH_DISTDIR GISBASE VERSION_NUMBER

############################################
# Cloning new manual pages into grass-stable/manuals/ (following the Python manual pages concept)
# - inject canonical URL therein to point to versioned manual page (avoiding "duplicate content" SEO punishment)
# see https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls

TARGETHTMLDIRSTABLE=$TARGETMAIN/grass-stable/manuals/
mkdir -p $TARGETHTMLDIRSTABLE $TARGETHTMLDIRSTABLE/addons
# cleanup from previous run
rm -rf /tmp/addons
\mv $TARGETHTMLDIRSTABLE/addons /tmp
rm -f $TARGETHTMLDIRSTABLE/*.*
(cd $TARGETHTMLDIRSTABLE ; rm -rf barscales colortables icons northarrows)
# clone manual pages
cp -rp $TARGETHTMLDIR/* $TARGETHTMLDIRSTABLE/
rm -rf $TARGETHTMLDIRSTABLE

@echoix echoix May 10, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So that means that temporarily, the docs will be unavailable if they are deleted before being rebuilt? (If the process doesn't finish its even worse).

What about having multiple "staging" folders, not served directly, that are like grass85-20260509205104 or something unique enough and the rebuild/unzipping of artifact is done in there, and when done, a symlink swaps what folder is served, atomically. Keep last 2-3 versions on the server. You could swap the symlink to a previous version if something goes wrong.

For not having these folders served but available to be served as symlink, there might be some friction with the configuration of some web servers, which might try to prevent you from serving something outside what is already publicly served.

@neteler neteler May 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally I agree. And would welcome implementation support.

rsync -a $TARGETHTMLDIR/ $TARGETHTMLDIRSTABLE/

@echoix echoix May 10, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a flag needed to propagate the deletions (if the line above is changed according to my other observation)?

@neteler neteler May 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to the hard cleanup a line below we here left out the (potentially dangerous) delete flag.


############################################
# SEO: inject canonical link into versioned manual pages (e.g, grass85/)
Expand Down Expand Up @@ -374,20 +291,12 @@ process_files "$TARGETHTMLDIR/libpython" "libpython/"
# SEO: "stable" manual pages (grass-stable/) is using canonical link

############################################
# create sitemaps to expand the hugo sitemap

# versioned manual:
python3 $HOME/src/grass$GMAJOR-addons/utils/create_manuals_sitemap.py --dir=/var/www/code_and_data/grass$GMAJOR$GMINOR/manuals/ --url=https://grass.osgeo.org/grass$GMAJOR$GMINOR/manuals/ -o
python3 $HOME/src/grass$GMAJOR-addons/utils/create_manuals_sitemap.py --dir=/var/www/code_and_data/grass$GMAJOR$GMINOR/manuals/addons/ --url=https://grass.osgeo.org/grass$GMAJOR$GMINOR/manuals/addons/ -o

# grass-stable manual:
python3 $HOME/src/grass$GMAJOR-addons/utils/create_manuals_sitemap.py --dir=/var/www/code_and_data/grass-stable/manuals/ --url=https://grass.osgeo.org/grass-stable/manuals/ -o
python3 $HOME/src/grass$GMAJOR-addons/utils/create_manuals_sitemap.py --dir=/var/www/code_and_data/grass-stable/manuals/addons/ --url=https://grass.osgeo.org/grass-stable/manuals/addons/ -o
# sitemaps to expand the hugo sitemap is generated by mkdocs GHA job

############################################
# cleanup
cd $GRASSBUILDDIR
$MYMAKE distclean > /dev/null || (echo "$0: an error occurred" ; exit 1)
$MYMAKE distclean > /dev/null 2>&1 || (echo "$0: an error occurred in final distclean" ; exit 1)
rm -rf lib/html/ lib/latex/ /tmp/addons

echo "Finished GRASS $VERSION $ARCH compilation."
Expand Down
6 changes: 4 additions & 2 deletions utils/cronjobs_osgeo_lxd/cron_grass_legacy_build_binaries.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,12 @@
# - injects in versioned manual the "canonical" to point to "stable" manual (as seen in the Python manual pages)

# Preparations, on server (neteler@grasslxd:$):
# - git clone the correct GRASS source code version into $HOME/src/
# - install dependencies:
# cd $HOME/src/releasebranch_7_8/ && git pull && sudo apt install $(cat .github/workflows/apt.txt)
# - install further dependencies:
# apt-get install texlive-latex-extra python3-sphinxcontrib.apidoc
# apt-get install texlive-latex-extra python3-sphinxcontrib.apidoc python3-sphinx-sitemap
# pip install sphinx-last-updated-by-git sphinx_material --break-system-packages # to avoid virtenv
# - run this script
# - one time only: cross-link code into web space on grasslxd server:
# cd /var/www/html/
Expand Down Expand Up @@ -161,7 +163,7 @@ echo "GRASS $VERSION compilation done"
#sh utils/module_synopsis.sh

#### generate developer stuff: pygrass docs + gunittest docs
# generate pyGRASS sphinx manual (in docs/html/libpython/)
# generate pyGRASS sphinx manual (in python/grass/docs/)
# including source code
$MYMAKE sphinxdoclib

Expand Down
6 changes: 4 additions & 2 deletions utils/cronjobs_osgeo_lxd/cron_grass_old_build_binaries.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,12 @@ exit 0
# - injects in versioned manual the "canonical" to point to "stable" manual (as seen in the Python manual pages)

# Preparations, on server (neteler@grasslxd:$):
# - git clone the correct GRASS source code version into $HOME/src/
# - install dependencies:
# cd $HOME/src/releasebranch_8_4/ && git pull && sudo apt install $(cat .github/workflows/apt.txt)
# - install further dependencies:
# apt-get install texlive-latex-extra python3-sphinxcontrib.apidoc
# apt-get install texlive-latex-extra python3-sphinxcontrib.apidoc python3-sphinx-sitemap
# pip install sphinx-last-updated-by-git sphinx_material --break-system-packages # to avoid virtenv
# - run this script
# - one time only: cross-link code into web space on grasslxd server:
# cd /var/www/html/
Expand Down Expand Up @@ -170,7 +172,7 @@ echo "GRASS $VERSION compilation done"
#sh utils/module_synopsis.sh

#### generate developer stuff: pygrass docs + gunittest docs
# generate pyGRASS sphinx manual (in docs/html/libpython/)
# generate pyGRASS sphinx manual (in python/grass/docs/)
# including source code
$MYMAKE sphinxdoclib

Expand Down
69 changes: 51 additions & 18 deletions utils/cronjobs_osgeo_lxd/cron_grass_preview_build_binaries.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@
# - generated the install scripts
# - generates the programmer's 8 HTML manual
# - note: the mkdocs based HTML manual pages are generated by GitHub Action
# - copies over generated manual pages to grass-devel/manuals/
# - injects in versioned manual the "canonical" to point to "devel" manual (as seen in the Python manual pages)

# Preparations, on server (neteler@grasslxd:$):

# - git clone the correct GRASS source code version into $HOME/src/
# - install dependencies:
# cd $HOME/src/main/ && git pull && sudo apt install $(cat .github/workflows/apt.txt)
# - install further dependencies (not needed for mkdocs!):
# apt-get install texlive-latex-extra python3-sphinxcontrib.apidoc
# - run this script
# - one time only: cross-link code into web space on grasslxd server:
# cd /var/www/html/
Expand Down Expand Up @@ -185,7 +185,9 @@ chmod -R a+r,g+w $TARGETPROGMAN/*
# bug in doxygen
(cd $TARGETPROGMAN/ ; ln -s index.html main.html)

############
# package the GRASS GIS package
cd $GRASSBUILDDIR
$MYMAKE bindist
if [ $? -ne 0 ] ; then
halt_on_error "make bindist."
Expand Down Expand Up @@ -235,19 +237,10 @@ sh $MAINDIR/cronjobs/compile_addons_git.sh $GMAJOR \
$MAINDIR/.grass$GMAJOR/addons \
$SOURCE/$BRANCH/bin.$ARCH/grass \
1
mkdir -p $TARGETHTMLDIR/addons/
# copy individual addon html files into one target dir if compiled addon
# has own dir e.g. $MAINDIR/.grass8/addons/db.join/ with bin/ docs/ etc/ scripts/
# subdir
for dir in `find $MAINDIR/.grass$GMAJOR/addons -maxdepth 1 -type d`; do
if [ -d $dir/docs/html ] ; then
if [ "$(ls -A $dir/docs/html/)" ]; then
for f in $dir/docs/html/*; do
echo "Skipping: cp $f $TARGETHTMLDIR/addons/"
done
fi
fi
done
### fetch manual from GH actions
# fetch artifacts for main (devel)
bash $HOME/cronjobs/gh_cli_download_artifact.sh $BRANCH
unzip -t /tmp/mkdocs-site_$BRANCH.zip && cd $TARGETHTMLDIR && rm -rf * && unzip -q /tmp/mkdocs-site_$BRANCH.zip && rm -f /tmp/mkdocs-site_$BRANCH.zip

# copy over logs from $MAINDIR/.grass$GMAJOR/addons/logs/
mkdir -p $TARGETMAIN/addons/grass$GMAJOR/logs/
Expand All @@ -257,17 +250,57 @@ cp -p $MAINDIR/.grass$GMAJOR/addons/logs/* $TARGETMAIN/addons/grass$GMAJOR/logs/
$SOURCE/$BRANCH/bin.$ARCH/grass --tmp-project EPSG:4326 --exec $MAINDIR/cronjobs/build-xml.py --build $MAINDIR/.grass$GMAJOR/addons
cp $MAINDIR/.grass$GMAJOR/addons/modules.xml $TARGETMAIN/addons/grass$GMAJOR/modules.xml

############################################
# Cloning new manual pages into grass-devel/manuals/ (following the Python manual pages concept)
# - inject canonical URL therein to point to versioned manual page (avoiding "duplicate content" SEO punishment)
# see https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls

TARGETHTMLDIRDEVEL=$TARGETMAIN/grass-devel/manuals/
# clone manual pages
rm -rf $TARGETHTMLDIRDEVEL
rsync -a $TARGETHTMLDIR/ $TARGETHTMLDIRDEVEL/
Comment on lines +260 to +261

@echoix echoix May 10, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar observations as for stable versions


############################################
# SEO: inject canonical link into versioned manual pages (e.g, grass86/)
# - cd back into folder of versioned HTML manual pages
# - run sed to replace an existing HTML header string in the upper part of the HTML file
# with itself + canonical link of devel version
# --> do this for core manual pages, addons, libpython, recursively

process_files() {
local dir="$1"
local prefix="$2"

find "$dir" -type f -name '*.html' | while IFS= read -r myfile; do
if ! grep -q 'link rel="canonical"' "$myfile"; then
manpage="$prefix$(basename ${myfile})"
sed -i -e "s:</head>:<link rel=\"canonical\" href=\"https\://grass.osgeo.org/grass-devel/manuals/$manpage\">\n</head>:g" ${myfile}
fi
done
}

cd "$TARGETHTMLDIR"
process_files "$TARGETHTMLDIR" ""
process_files "$TARGETHTMLDIR/addons" "addons/"
process_files "$TARGETHTMLDIR/libpython" "libpython/"

# SEO: "devel" manual pages (grass-devel/) is using canonical link

############################################
# sitemaps to expand the hugo sitemap is generated by mkdocs GHA job

############################################
# cleanup
cd $GRASSBUILDDIR
$MYMAKE distclean > /dev/null || (echo "$0: an error occurred" ; exit 1)
$MYMAKE distclean > /dev/null 2>&1 || (echo "$0: an error occurred in final distclean" ; exit 1)
rm -rf lib/html/ lib/latex/ /tmp/addons

echo "Finished GRASS $VERSION $ARCH compilation."
echo "Written to: $TARGETDIR"
echo "Copied HTML ${GVERSION} progman to https://grass.osgeo.org/programming${GVERSION}"
echo "Copied HTML ${GVERSION} manual to https://grass.osgeo.org/grass${VERSION}/manuals/ (with canonical in metadata)"
echo "Copied pygrass progman ${GVERSION} to https://grass.osgeo.org/grass${VERSION}/manuals/libpython/ (with canonical in metadata)"
echo "Copied Addons ${GVERSION} to https://grass.osgeo.org/grass${VERSION}/manuals/addons/ (with canonical in metadata)"
## echo "Copied HTML ${GVERSION} progman to https://grass.osgeo.org/programming${GVERSION}"
echo "Copied HTML devel manual to https://grass.osgeo.org/grass-devel/manuals/"

exit 0
4 changes: 2 additions & 2 deletions utils/cronjobs_osgeo_lxd/cron_job_list_grass
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@

# run at 00:30:00 and 12:30:00 each day: fetches code from GitHub and builds it with hugo (target dir: /var/www/html/)
30 */12 * * * nice bash /home/neteler/cronjobs/hugo_clean_and_update_job.sh
# run at 00:40:00 and 12:40:00 each day:
40 */12 * * * nice bash /home/neteler/cronjobs/fetch_unpack_manual_GHA.sh > /home/neteler/fetch_unpack_manual_GHA.log 2>&1
# run at 00:40:00 and 12:40:00 each day (disabled May 2026, integrated in the respective build scripts)
#40 */12 * * * nice bash /home/neteler/cronjobs/fetch_unpack_manual_GHA.sh > /home/neteler/fetch_unpack_manual_GHA.log 2>&1

# weekly source snapshots (target dir: /var/www/code_and_data/)
# legacy
Expand Down
32 changes: 0 additions & 32 deletions utils/cronjobs_osgeo_lxd/fetch_unpack_manual_GHA.sh
View file
Open in desktop

This file was deleted.

Loading