[QGIS-Developer] Reading OSM data from .osm file

[matteo]

Hi all,

downloading an osm file from https://www.openstreetmap.org and loading
it in QGIS results in perfect spatial positions but the attribute data
are read wrong.

Wrong means that there are only 2 columns: "osm_id" with correct numbers
of feature ids and another nameless column that contains all the other
information like: "crossing"=>"traffic_signals","highway"=>"crossing"

Seems that the parser is not working correctly?

To reproduce the problem:

1. https://www.openstreetmap.org
2. Click on Export in the top left corner
3. click on Export in the new tab
4. open the osm file in QGIS

Someone confirms? Should I open a ticket?

Thanks to all

Matteo

[Dr. Marco Lechner]

Hi Matteo,

I guess you are missing an understanding for OSMs data model. OSM is
very flexible in adding attributes to a feature by adding key/value
pairs. If this is to be represented in a GIS data away (table) you have
to convert the data importing just relevant keys (each becoming a
column), which results in loss of some attributes, or you can use
something like hstore in postgresql [1] meaning you store the set of
key/value pairs (or dictionary in python-terms) in one column. This is
lossless and seems to be what is used here.

Marco

[1] https://www.postgresql.org/docs/9.3/static/hstore.html

Post by matteo
Hi all,
downloading an osm file from https://www.openstreetmap.org and loading
it in QGIS results in perfect spatial positions but the attribute data
are read wrong.
Wrong means that there are only 2 columns: "osm_id" with correct numbers
of feature ids and another nameless column that contains all the other
information like: "crossing"=>"traffic_signals","highway"=>"crossing"
Seems that the parser is not working correctly?
1. https://www.openstreetmap.org
2. Click on Export in the top left corner
3. click on Export in the new tab
4. open the osm file in QGIS
Someone confirms? Should I open a ticket?
Thanks to all
Matteo
_______________________________________________
QGIS-Developer mailing list
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[matteo]

Hi Marco,

thanks for the answer and the explanation.

Post by Dr. Marco Lechner
I guess you are missing an understanding for OSMs data model. OSM is
very flexible in adding attributes to a feature by adding key/value
pairs. If this is to be represented in a GIS data away (table) you have
to convert the data importing just relevant keys (each becoming a
column), which results in loss of some attributes, or you can use
something like hstore in postgresql [1] meaning you store the set of
key/value pairs (or dictionary in python-terms) in one column. This is
lossless and seems to be what is used here.

well I think that even if lossless the information are pretty much
useless also.

What I'm saying is that the old OSM core provider of QGIS 2 (download
osm and convert osm to SpatiaLite) and the current QuickOSM plugin (both
QGIS 2 and QGIS 3) stores all the information as well, having maybe a
lot of columns empty, but still usable.

Cheers

Matteo

[Etienne Trimaille]

It's the purpose of the QuickOSM default parser. It will not lose any
key/value, so you will have one column for each keys.

But if you load your file without plugins and not with the OSM core
provider in QGIS 2, I'm surprise that you got the "highway" key on the
"other_tags" field. I suspect you modified your own osmconf.ini.
According to the default file, "highway" must have its own field:
https://github.com/OSGeo/gdal/blob/master/gdal/data/osmconf.ini#L48

You can still use the default parser and use the whitelist field in
QuickOSM QGIS 2. It will parse only these OSM keys (separated by a comma).
If you have your own OSM file on your computer, when you open it with
QuickOSM (2 and 3), you can provide your custom OSMConf.ini file which
specify which OSM keys you want to parse.

Post by matteo
Hi Marco,
thanks for the answer and the explanation.

Post by Dr. Marco Lechner
I guess you are missing an understanding for OSMs data model. OSM is
very flexible in adding attributes to a feature by adding key/value
pairs. If this is to be represented in a GIS data away (table) you have
to convert the data importing just relevant keys (each becoming a
column), which results in loss of some attributes, or you can use
something like hstore in postgresql [1] meaning you store the set of
key/value pairs (or dictionary in python-terms) in one column. This is
lossless and seems to be what is used here.

well I think that even if lossless the information are pretty much
useless also.
What I'm saying is that the old OSM core provider of QGIS 2 (download
osm and convert osm to SpatiaLite) and the current QuickOSM plugin (both
QGIS 2 and QGIS 3) stores all the information as well, having maybe a
lot of columns empty, but still usable.
Cheers
Matteo
_______________________________________________
QGIS-Developer mailing list
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[matteo]

Hi Etienne,

wait maybe we are confusing providers ;)

QuickOSM parser works great. So far I never had a problem when
downloading data with your plugin.

But if the .osm file is downloaded outside QGIS (e.g. directly on
openstreetmap.com) the parser (GDAL?!) makes columns collapsing.

I attached a screenshot of the only 2 columns I have from the point
layer loaded from the osm file (center of Rome city). So except for
"osm_id" **all** the other tags are concatenated in the nameless second
column.

I really don't know is I'm missing something, but I thought that the
workflow was pretty straightforward.

Thanks!

Matteo

[Etienne Trimaille]

When you have your .osm file on your computer (eg downloaded from outside
QGIS), you can open it with:
* QuickOSM with the default parser, each keys will have their own fields.
* QuickOSM with your custom osmconf.ini file. The attribute table is
defined according to your osmconf file.
* OSM core importer in QGIS 2, not sure about the final attribute table.
* QGIS, loading a normal vector file. The osmconf file on your computer
will be used to define the attribute table.

So it seems you are using the last option, right? So maybe you modified the
default osmconf file?

Post by matteo
Hi Etienne,
wait maybe we are confusing providers ;)
QuickOSM parser works great. So far I never had a problem when
downloading data with your plugin.
But if the .osm file is downloaded outside QGIS (e.g. directly on
openstreetmap.com) the parser (GDAL?!) makes columns collapsing.
I attached a screenshot of the only 2 columns I have from the point
layer loaded from the osm file (center of Rome city). So except for
"osm_id" **all** the other tags are concatenated in the nameless second
column.
I really don't know is I'm missing something, but I thought that the
workflow was pretty straightforward.
Thanks!
Matteo

[matteo]

Hey,

Post by Etienne Trimaille
When you have your .osm file on your computer (eg downloaded from
* QuickOSM with the default parser, each keys will have their own fields.
* QuickOSM with your custom osmconf.ini file. The attribute table is
defined according to your osmconf file.
* OSM core importer in QGIS 2, not sure about the final attribute table.
* QGIS, loading a normal vector file. The osmconf file on your computer
will be used to define the attribute table.
So it seems you are using the last option, right? So maybe you modified
the default osmconf file?

it seems so. Never touched the osmconf file (almost a fresh Debian Sid
OS). It just seems strange to me that the "default" parser (the last one
of the list) is lossy (meaning that many columns can collapse into a
single not usable one).

Anyway: QuickOSM parser works like a charm ;)

Thanks to all

Matteo

[DelazJ]

Hi,

Etienne, where does the osmconf file come from? Is it a default file in
QGIS (v3) installation or something added when installing the QuickOSM
plugin?
I mean should the aforementioned option 4 have returned the data in a
""flat"" shape (a key <-> a field), as does QuickOSM by default?

Thanks,
Harrissou

Post by matteo
Hey,

Post by Etienne Trimaille
When you have your .osm file on your computer (eg downloaded from
* QuickOSM with the default parser, each keys will have their own fields.
* QuickOSM with your custom osmconf.ini file. The attribute table is
defined according to your osmconf file.
* OSM core importer in QGIS 2, not sure about the final attribute table.
* QGIS, loading a normal vector file. The osmconf file on your computer
will be used to define the attribute table.
So it seems you are using the last option, right? So maybe you modified
the default osmconf file?

it seems so. Never touched the osmconf file (almost a fresh Debian Sid
OS). It just seems strange to me that the "default" parser (the last one
of the list) is lossy (meaning that many columns can collapse into a
single not usable one).
Anyway: QuickOSM parser works like a charm ;)
Thanks to all
Matteo
_______________________________________________
QGIS-Developer mailing list
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[Etienne Trimaille]

For me, I have it in /usr/share/gdal/2.2/osmconf.ini
It's by default with gdal. Only a few columns should be created by default:
https://github.com/OSGeo/gdal/blob/master/gdal/data/osmconf.ini
<https://github.com/OSGeo/gdal/blob/master/gdal/data/osmconf.ini#L48>
http://www.gdal.org/drv_osm.html how to use this file.

Post by DelazJ
Hi,
Etienne, where does the osmconf file come from? Is it a default file in
QGIS (v3) installation or something added when installing the QuickOSM
plugin?
I mean should the aforementioned option 4 have returned the data in a
""flat"" shape (a key <-> a field), as does QuickOSM by default?
Thanks,
Harrissou

Post by Etienne Trimaille
Hey,

Post by Etienne Trimaille
When you have your .osm file on your computer (eg downloaded from
* QuickOSM with the default parser, each keys will have their own

fields.

Post by Etienne Trimaille
* QuickOSM with your custom osmconf.ini file. The attribute table is
defined according to your osmconf file.
* OSM core importer in QGIS 2, not sure about the final attribute table.
* QGIS, loading a normal vector file. The osmconf file on your computer
will be used to define the attribute table.
So it seems you are using the last option, right? So maybe you modified
the default osmconf file?

it seems so. Never touched the osmconf file (almost a fresh Debian Sid
OS). It just seems strange to me that the "default" parser (the last one
of the list) is lossy (meaning that many columns can collapse into a
single not usable one).
Anyway: QuickOSM parser works like a charm ;)
Thanks to all
Matteo
_______________________________________________
QGIS-Developer mailing list
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[Etienne Trimaille]

I could replicate your issue too on QGIS 3.2 Matteo.
Not always, maybe depending on the geometry or how many layer you load at
the same time:
Maybe linked to https://issues.qgis.org/issues/19477 (I got some different
behaviors according to the number of layer loaded) and
https://issues.qgis.org/issues/19461

Post by matteo
Hey,

Post by Etienne Trimaille
When you have your .osm file on your computer (eg downloaded from
* QuickOSM with the default parser, each keys will have their own fields.
* QuickOSM with your custom osmconf.ini file. The attribute table is
defined according to your osmconf file.
* OSM core importer in QGIS 2, not sure about the final attribute table.
* QGIS, loading a normal vector file. The osmconf file on your computer
will be used to define the attribute table.
So it seems you are using the last option, right? So maybe you modified
the default osmconf file?

it seems so. Never touched the osmconf file (almost a fresh Debian Sid
OS). It just seems strange to me that the "default" parser (the last one
of the list) is lossy (meaning that many columns can collapse into a
single not usable one).
Anyway: QuickOSM parser works like a charm ;)
Thanks to all
Matteo