Martin Dobias
2018-10-11 15:17:41 UTC
Hi all
Recently I needed transformation between points from high-accuracy GPS
to British National Grid (EPSG:27700) which involves datum shift (WGS
84 to OSGB 1936) using OSTN15 grid.
The process to do that with QGIS is currently... well... really
clumsy. One needs to:
1. manually download grids from the web and extract .gsb file to Proj
data directory
2. manually add an entry to tbl_datum_transform table in srs.db in
QGIS installation
There are two main problems with that:
- both steps need adjustment of QGIS installation - not everyone has
admin privileges
- doing manual changes to srs.db is just too much to ask from ordinary user
I understand that it is hard to ship datum shift grids with QGIS due
to various issues (their size and amount, sometimes licensing) and
that's fine. But we should think about how to make it easier to use
them. There is a QGIS plugin [1] that provides a set of algs for
Processing that call ogr2ogr with the right grid files and download
them if they are not yet available - however there is no integration
with QGIS CRS database, so any other transforms inside QGIS can't make
use of that.
Note for those not familiar with srs.db: it is a sqlite3 db living in
QGIS source code which has been generated at some point from GDAL data
files. Over time it has gathered some manual ad-hoc updates of
parameters. Then upon QGIS build/installation we run "crssync" utility
to parse current GDAL data files again and update/insert CRS and datum
shift parameters (see QgsCoordinateReferenceSystem::syncDatabase()).
My suggestions for QGIS:
1. any manual additions to srs.db should be listed as SQL commands
because it is difficult to track changes to srs.db which is a binary
blob (sqlite db). During the build of QGIS we would create final
srs.db
2. rather than manually adding entries about datum shift grids to
tbl_datum_transform table (which is generated from GDAL's
datum_shift.csv [2]) and contains parameters of 3/7 parameter
transform between datums, there should be a new table just for grid
files, ideally with links where they can be downloaded. (fun fact:
names of grid files in tbl_datum_transform table are stored in a
numeric column but because this is sqlite it just works)
3. it should be possible to add custom entries about datum shift grids
by users (just like it is possible to add custom CRS) - that is, such
table should be also in user's profile (in qgis.db)
My suggestions for Proj:
1. it should be possible to specify multiple data directories with
PROJ_LIB env variable, so that it is possible to use grids also from
user's home directory (not just the read-only system directory)
2. there could be a database similar to datum_shift.csv [2] that would
list well known datum shift grid files, so that QGIS does not need to
maintain its database and just pull that information out of Proj -
other projects could benefit from that as well.
I am aware of the fact that there is a big refactoring in Proj going
on [3] but I am not sure if any of the above suggestions are also in
the plans...
I would be happy to help implementing some of the above, but I would
like to first hear opinion of others - maybe I got something wrong and
maybe there are better ways to move forward. I hope gurus like
Kristian or Even read this and can offer some of their wisdom too :-)
Cheers
Martin
[1] https://github.com/NaturalGIS/naturalgis_ntv2_transformations
[2] https://github.com/OSGeo/gdal/blob/master/gdal/data/datum_shift.csv
[3] https://gdalbarn.com/
Recently I needed transformation between points from high-accuracy GPS
to British National Grid (EPSG:27700) which involves datum shift (WGS
84 to OSGB 1936) using OSTN15 grid.
The process to do that with QGIS is currently... well... really
clumsy. One needs to:
1. manually download grids from the web and extract .gsb file to Proj
data directory
2. manually add an entry to tbl_datum_transform table in srs.db in
QGIS installation
There are two main problems with that:
- both steps need adjustment of QGIS installation - not everyone has
admin privileges
- doing manual changes to srs.db is just too much to ask from ordinary user
I understand that it is hard to ship datum shift grids with QGIS due
to various issues (their size and amount, sometimes licensing) and
that's fine. But we should think about how to make it easier to use
them. There is a QGIS plugin [1] that provides a set of algs for
Processing that call ogr2ogr with the right grid files and download
them if they are not yet available - however there is no integration
with QGIS CRS database, so any other transforms inside QGIS can't make
use of that.
Note for those not familiar with srs.db: it is a sqlite3 db living in
QGIS source code which has been generated at some point from GDAL data
files. Over time it has gathered some manual ad-hoc updates of
parameters. Then upon QGIS build/installation we run "crssync" utility
to parse current GDAL data files again and update/insert CRS and datum
shift parameters (see QgsCoordinateReferenceSystem::syncDatabase()).
My suggestions for QGIS:
1. any manual additions to srs.db should be listed as SQL commands
because it is difficult to track changes to srs.db which is a binary
blob (sqlite db). During the build of QGIS we would create final
srs.db
2. rather than manually adding entries about datum shift grids to
tbl_datum_transform table (which is generated from GDAL's
datum_shift.csv [2]) and contains parameters of 3/7 parameter
transform between datums, there should be a new table just for grid
files, ideally with links where they can be downloaded. (fun fact:
names of grid files in tbl_datum_transform table are stored in a
numeric column but because this is sqlite it just works)
3. it should be possible to add custom entries about datum shift grids
by users (just like it is possible to add custom CRS) - that is, such
table should be also in user's profile (in qgis.db)
My suggestions for Proj:
1. it should be possible to specify multiple data directories with
PROJ_LIB env variable, so that it is possible to use grids also from
user's home directory (not just the read-only system directory)
2. there could be a database similar to datum_shift.csv [2] that would
list well known datum shift grid files, so that QGIS does not need to
maintain its database and just pull that information out of Proj -
other projects could benefit from that as well.
I am aware of the fact that there is a big refactoring in Proj going
on [3] but I am not sure if any of the above suggestions are also in
the plans...
I would be happy to help implementing some of the above, but I would
like to first hear opinion of others - maybe I got something wrong and
maybe there are better ways to move forward. I hope gurus like
Kristian or Even read this and can offer some of their wisdom too :-)
Cheers
Martin
[1] https://github.com/NaturalGIS/naturalgis_ntv2_transformations
[2] https://github.com/OSGeo/gdal/blob/master/gdal/data/datum_shift.csv
[3] https://gdalbarn.com/