Discussion:
Improving ftools geoprocessing tools
(too old to reply)
João Gaspar
2015-10-22 11:30:40 UTC
Permalink
Dear QGIS Developers,

I'm writing on behalf of my employer, a company that has mixed,
proprietary and open source (especially QGIS of course) gis
infrastructure, with qgis being adopted each day a little more.
Actually in our migration path we have found a road blocker: the very
bad performances (and sometimes results) of most of geoprocessing
tools in the vector menu. Yes I know that there are alternatives in
the Processing toolbox, but that is not an option for most of our very
basic users: they need to do simple operations (clips, unions,
intersections, etc.) with usually very large/detailed vectors (usually
shapefiles). In this scenario QGIS usually chokes or is at best very
slow (compared to other gis packages, the ones we are trying to get
rid of).

What can be done to improve this tools? Rewriting the tools to improve
efficiency and accuracy?

We available some funds: this is the first time I convince people
above me to put some money in QGIS development, and a decision would
need to me made until the end of November tops, otherwise money will
be used otherwise.

Please leave advice and feedback.


Regards
João Gaspar
Bernd Vogelgesang
2015-10-22 12:06:04 UTC
Permalink
Post by João Gaspar
Dear QGIS Developers,
I'm writing on behalf of my employer, a company that has mixed,
proprietary and open source (especially QGIS of course) gis
infrastructure, with qgis being adopted each day a little more.
Actually in our migration path we have found a road blocker: the very
bad performances (and sometimes results) of most of geoprocessing
tools in the vector menu. Yes I know that there are alternatives in
the Processing toolbox, but that is not an option for most of our very
basic users: they need to do simple operations (clips, unions,
intersections, etc.) with usually very large/detailed vectors (usually
shapefiles). In this scenario QGIS usually chokes or is at best very
slow (compared to other gis packages, the ones we are trying to get
rid of).
What can be done to improve this tools? Rewriting the tools to improve
efficiency and accuracy?
We available some funds: this is the first time I convince people
above me to put some money in QGIS development, and a decision would
need to me made until the end of November tops, otherwise money will
be used otherwise.
Please leave advice and feedback.
Regards
João Gaspar
Hi,
I'm not a developer, but a simple user following this list.

My point of view is, that the ftools are just a relict from the past since
the processing toolbox was introduced.
I'm glad that people step up for funding QGIS, but in this particular
case, you would better invest your money and time in lifting your "basic"
users to a little more advanced users and show them how to use the
processing toolbox properly.

As far as I know, the toolbox is nearly feature complete with the ftools
functions, and most of them are so much faster and/or versatile, that I
did not touch the ftools for ages.
Prepare some centrally stored models for reoccuring use cases, and have
happy users ;)

Just my two cents
Bernd
--
Bernd Vogelgesang
Siedlerstraße 2
91083 Baiersdorf/Igelsdorf
Tel: 09133-825374
Matthias Kuhn
2015-10-22 12:28:47 UTC
Permalink
Hi João,

Nice that there is interest in improving the analysis part of QGIS.

There was recently a similar discussion with Bernhard Ströbl who also
put some effort into improving the dissolve tool (although it turned out
to be complex and will probably only be merged in 2.14).

In the discussion there, the need for coexistence of these two suits
offering the same tools was questioned. In particular, maintaining twice
the same tool in QGIS is an extra effort in maintenance and there is no
advantage visible.

It would be nice to plan how to resolv this situation, in particular
could you answer the following questions:

* Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)
* Are the processing toolbox tools enough performant for your needs? If
not, which ones require attention? It would be good to have a list.

If the first one is a problem, I think it would be an approach to let
system administrators define "favorite algorithms" (or similar) which
would then be offered as a shortcut in the menu.

If the second one applies, then we already have proven the point that
maintaining the same algorithm twice is silly and the effort should
rather be put into unifying these tools instead of rewriting duplicate
code. I will probably soon be working on a test suite for processing
which should ensure long-term quality assurance for the algorithms
there. Another advantage of consolidating the two code bases.

I hope that it's possible for your company to consider different
approaches for this problem to get to a long-term enhancement.

Best regards,
Matthias
Victor Olaya
2015-10-22 12:59:25 UTC
Permalink
Post by Matthias Kuhn
* Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)
If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?

And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.

Regards
Andreas Neumann
2015-10-22 13:13:30 UTC
Permalink
If we'd replace the ftools (vector and raster menu) through processing.
What would replace them? Would we do research and see which of the
alternative processing providers provides best speed/reliability - if
there are multiple versions? It may be, that for Algorithm A, Saga works
best, whereas for algorithm B GRASS is better and for C GDAL or the QGIS
builtin in mechanism.

Andreas
Post by Victor Olaya
Post by Matthias Kuhn
* Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)
If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?
And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.
Regards
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Bernhard Ströbl
2015-10-22 13:26:07 UTC
Permalink
IMHO all ftools should be replaced by native QGIS processing algorithms.
For "simple" users it may be not feasible (or not allowed) to install
additional software. Therefore native QGIS algorithms should be improved
to be fast, robust and offering choices the users need (see recent
discussion on dissolve)

Bernhard
Post by Andreas Neumann
If we'd replace the ftools (vector and raster menu) through processing.
What would replace them? Would we do research and see which of the
alternative processing providers provides best speed/reliability - if
there are multiple versions? It may be, that for Algorithm A, Saga works
best, whereas for algorithm B GRASS is better and for C GDAL or the QGIS
builtin in mechanism.
Andreas
Post by Victor Olaya
Post by Matthias Kuhn
* Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)
If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?
And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.
Regards
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
__________ Information from ESET Mail Security, version of virus signature database 12448 (20151022) __________

The message was checked by ESET Mail Security.
http://www.eset.com
Andreas Neumann
2015-10-22 13:48:58 UTC
Permalink
Bernhard,

I agree that these tools should be QGIS algorithms - maybe even ported
from another GIS (if feasible).

One major annoyance with fTools and some processing algorithms is that
reliance on ESRI shapefile as output. That way I always loose my columns
- or they are truncated to be meaningless. For that reason I often don't
use QGIS, but FME for analysis. Or try to do the processing with SQL in
Postgis.

If we renovate these algorithms in the vector menu we'd have to make
sure that the output could be any chosen vector format, or at least
supports geopackage/spatialite/postgres as output.

Andreas
Post by Bernhard Ströbl
IMHO all ftools should be replaced by native QGIS processing
algorithms. For "simple" users it may be not feasible (or not allowed)
to install additional software. Therefore native QGIS algorithms
should be improved to be fast, robust and offering choices the users
need (see recent discussion on dissolve)
Bernhard
Post by Andreas Neumann
If we'd replace the ftools (vector and raster menu) through processing.
What would replace them? Would we do research and see which of the
alternative processing providers provides best speed/reliability - if
there are multiple versions? It may be, that for Algorithm A, Saga works
best, whereas for algorithm B GRASS is better and for C GDAL or the QGIS
builtin in mechanism.
Andreas
Post by Victor Olaya
Post by Matthias Kuhn
* Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)
If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?
And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.
Regards
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
__________ Information from ESET Mail Security, version of virus
signature database 12448 (20151022) __________
The message was checked by ESET Mail Security.
http://www.eset.com
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Bernhard Ströbl
2015-10-22 14:08:12 UTC
Permalink
Hi Andreas,

AFAIK processing uses shp for in between results in a model, too. So if
you build a model you loose your fields, no matter which output format
you choose. IMHO it should therefore be considered to change
processing's data source. Maybe SpatialLite is an option? Users can
still export to shp if needed.

Bernhard
Post by Andreas Neumann
Bernhard,
I agree that these tools should be QGIS algorithms - maybe even ported
from another GIS (if feasible).
One major annoyance with fTools and some processing algorithms is that
reliance on ESRI shapefile as output. That way I always loose my columns
- or they are truncated to be meaningless. For that reason I often don't
use QGIS, but FME for analysis. Or try to do the processing with SQL in
Postgis.
If we renovate these algorithms in the vector menu we'd have to make
sure that the output could be any chosen vector format, or at least
supports geopackage/spatialite/postgres as output.
Andreas
Post by Bernhard Ströbl
IMHO all ftools should be replaced by native QGIS processing
algorithms. For "simple" users it may be not feasible (or not allowed)
to install additional software. Therefore native QGIS algorithms
should be improved to be fast, robust and offering choices the users
need (see recent discussion on dissolve)
Bernhard
Post by Andreas Neumann
If we'd replace the ftools (vector and raster menu) through processing.
What would replace them? Would we do research and see which of the
alternative processing providers provides best speed/reliability - if
there are multiple versions? It may be, that for Algorithm A, Saga works
best, whereas for algorithm B GRASS is better and for C GDAL or the QGIS
builtin in mechanism.
Andreas
Post by Victor Olaya
Post by Matthias Kuhn
* Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)
If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?
And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.
Regards
__________ Information from ESET Mail Security, version of virus signature database 12448 (20151022) __________

The message was checked by ESET Mail Security.
http://www.eset.com
Victor Olaya
2015-10-22 14:14:10 UTC
Permalink
shp is the default format for temp files, and as such it is used in
the modeler for intermediate results. That is, however, very easy to
change, so if we agree on any other format, it can be changed and set
as the default for Processing
Post by Bernhard Ströbl
Hi Andreas,
AFAIK processing uses shp for in between results in a model, too. So if you
build a model you loose your fields, no matter which output format you
choose. IMHO it should therefore be considered to change processing's data
source. Maybe SpatialLite is an option? Users can still export to shp if
needed.
Bernhard
Post by Andreas Neumann
Bernhard,
I agree that these tools should be QGIS algorithms - maybe even ported
from another GIS (if feasible).
One major annoyance with fTools and some processing algorithms is that
reliance on ESRI shapefile as output. That way I always loose my columns
- or they are truncated to be meaningless. For that reason I often don't
use QGIS, but FME for analysis. Or try to do the processing with SQL in
Postgis.
If we renovate these algorithms in the vector menu we'd have to make
sure that the output could be any chosen vector format, or at least
supports geopackage/spatialite/postgres as output.
Andreas
Post by Bernhard Ströbl
IMHO all ftools should be replaced by native QGIS processing
algorithms. For "simple" users it may be not feasible (or not allowed)
to install additional software. Therefore native QGIS algorithms
should be improved to be fast, robust and offering choices the users
need (see recent discussion on dissolve)
Bernhard
Post by Andreas Neumann
If we'd replace the ftools (vector and raster menu) through processing.
What would replace them? Would we do research and see which of the
alternative processing providers provides best speed/reliability - if
there are multiple versions? It may be, that for Algorithm A, Saga works
best, whereas for algorithm B GRASS is better and for C GDAL or the QGIS
builtin in mechanism.
Andreas
Post by Victor Olaya
Post by Matthias Kuhn
* Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)
If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?
And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.
Regards
__________ Information from ESET Mail Security, version of virus signature
database 12448 (20151022) __________
The message was checked by ESET Mail Security.
http://www.eset.com
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Bernhard Ströbl
2015-10-22 14:21:41 UTC
Permalink
What what be a suitable format? It should
1) support all geometry types needed
2) be fast
3) support spatial indices?
4) support long (unlimited) field names
5) support all necessary field types
6) store its data on local hd

anything else?

Bernhard
Post by Victor Olaya
shp is the default format for temp files, and as such it is used in
the modeler for intermediate results. That is, however, very easy to
change, so if we agree on any other format, it can be changed and set
as the default for Processing
Post by Bernhard Ströbl
Hi Andreas,
AFAIK processing uses shp for in between results in a model, too. So if you
build a model you loose your fields, no matter which output format you
choose. IMHO it should therefore be considered to change processing's data
source. Maybe SpatialLite is an option? Users can still export to shp if
needed.
Bernhard
Post by Andreas Neumann
Bernhard,
I agree that these tools should be QGIS algorithms - maybe even ported
from another GIS (if feasible).
One major annoyance with fTools and some processing algorithms is that
reliance on ESRI shapefile as output. That way I always loose my columns
- or they are truncated to be meaningless. For that reason I often don't
use QGIS, but FME for analysis. Or try to do the processing with SQL in
Postgis.
If we renovate these algorithms in the vector menu we'd have to make
sure that the output could be any chosen vector format, or at least
supports geopackage/spatialite/postgres as output.
Andreas
Post by Bernhard Ströbl
IMHO all ftools should be replaced by native QGIS processing
algorithms. For "simple" users it may be not feasible (or not allowed)
to install additional software. Therefore native QGIS algorithms
should be improved to be fast, robust and offering choices the users
need (see recent discussion on dissolve)
Bernhard
Post by Andreas Neumann
If we'd replace the ftools (vector and raster menu) through processing.
What would replace them? Would we do research and see which of the
alternative processing providers provides best speed/reliability - if
there are multiple versions? It may be, that for Algorithm A, Saga works
best, whereas for algorithm B GRASS is better and for C GDAL or the QGIS
builtin in mechanism.
Andreas
Post by Victor Olaya
Post by Matthias Kuhn
* Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)
If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?
And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.
Regards
__________ Information from ESET Mail Security, version of virus signature database 12448 (20151022) __________

The message was checked by ESET Mail Security.
http://www.eset.com
Bernd Vogelgesang
2015-10-22 14:35:05 UTC
Permalink
Post by Victor Olaya
shp is the default format for temp files, and as such it is used in
the modeler for intermediate results. That is, however, very easy to
change, so if we agree on any other format, it can be changed and set
as the default for Processing
+1000 Yes, please, get rid of this pain in the a.. as soon as possible and
use SpatiaLite e.g..
Creating these files might take considerably longer and they consume more
space, but compared to the time I spend refactoring column names all the
time and finding workarounds, this is nothing.
It's also one of the reasons why I do my heavy lifting work in R nowadays.

Cheers
Bernd
Post by Victor Olaya
Post by Bernhard Ströbl
Hi Andreas,
AFAIK processing uses shp for in between results in a model, too. So if you
build a model you loose your fields, no matter which output format you
choose. IMHO it should therefore be considered to change processing's data
source. Maybe SpatialLite is an option? Users can still export to shp if
needed.
Bernhard
Post by Andreas Neumann
Bernhard,
I agree that these tools should be QGIS algorithms - maybe even ported
from another GIS (if feasible).
One major annoyance with fTools and some processing algorithms is that
reliance on ESRI shapefile as output. That way I always loose my columns
- or they are truncated to be meaningless. For that reason I often don't
use QGIS, but FME for analysis. Or try to do the processing with SQL in
Postgis.
If we renovate these algorithms in the vector menu we'd have to make
sure that the output could be any chosen vector format, or at least
supports geopackage/spatialite/postgres as output.
Andreas
Post by Bernhard Ströbl
IMHO all ftools should be replaced by native QGIS processing
algorithms. For "simple" users it may be not feasible (or not allowed)
to install additional software. Therefore native QGIS algorithms
should be improved to be fast, robust and offering choices the users
need (see recent discussion on dissolve)
Bernhard
Post by Andreas Neumann
If we'd replace the ftools (vector and raster menu) through processing.
What would replace them? Would we do research and see which of the
alternative processing providers provides best speed/reliability - if
there are multiple versions? It may be, that for Algorithm A, Saga works
best, whereas for algorithm B GRASS is better and for C GDAL or the QGIS
builtin in mechanism.
Andreas
Post by Victor Olaya
Post by Matthias Kuhn
* Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)
If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?
And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.
Regards
__________ Information from ESET Mail Security, version of virus signature
database 12448 (20151022) __________
The message was checked by ESET Mail Security.
http://www.eset.com
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
--
Bernd Vogelgesang
Siedlerstraße 2
91083 Baiersdorf/Igelsdorf
Tel: 09133-825374
Victor Olaya
2015-10-22 14:07:19 UTC
Permalink
If we'd replace the ftools (vector and raster menu) through processing. What
would replace them? Would we do research and see which of the alternative
processing providers provides best speed/reliability - if there are multiple
versions? It may be, that for Algorithm A, Saga works best, whereas for
algorithm B GRASS is better and for C GDAL or the QGIS builtin in mechanism.
SAGA algorithms are pretty fast, and we wont probably get that
performance unless we move the logic of algorithms to the C++ core (as
Alex suggested). However, I wouldn't put the SAGA algorithm to replace
the ftools menu, for the reason that you mentioned: SAGA only handles
shapefiles. I think that it would be better to get a good performance
(even if it is not that good when compared to GRASS or SAGA), but have
the support for all formats. Processing will give you that support,
converting files as needed, but in the end, it has to pass through a
shapefile, and we have the problem with field names...
Mathieu Pellerin
2015-10-22 13:17:23 UTC
Permalink
QGIS 3.0 seems to be a good moment/opportunity to terminate ftools and
replace the menu items to be shortcuts to processing algorithms.

M
Post by Victor Olaya
Post by Matthias Kuhn
* Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)
If what you need is to have the tools in a menu, we have been
discussing that before, and the idea is to mantain the same menu
structure if the users wants that, but have the algorithms based on
Processing to avoid redundant (and more difficult to mantain) code. As
I said, this has already been discussed, but it is not yet
implemented. Maybe your funding could help us do it?
And if algorithms are not performant enough, it's clearly better to
put your effort in the Processing ones (which, in general, are just a
copy of the ftools ones), and have them exposed through the menus, as
explained above.
Regards
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Giovanni Manghi
2015-10-22 12:38:47 UTC
Permalink
Hi Matthias,

the issues are the following:


there is whole universe of very basic users that need to do classic
geoprocessing operations (clips, unions, etc.) sometimes also with
large/dense geometries, possibly with geometry errors. This users are
usually not able to handle (or willingly to) the possible problems of
using a toolbox based on other software. For them the qgis
geoprocessing tools would be enough (now, as a separet menu as now or
only in processing this is not an issue).

problem is that when using large/dense inputs this qgis tools are
usually very slow, especially when compared to the proprietary
software many are trying to leave. In other worst cases the results
are wrong, but not in a way that is openly wrong, making the usability
of such tools even more difficult.

so said that I understand perfectly what João says, and I would say
that is time to the action, especially considering that there is an
offer on the table.

cheers

-- g --
Post by Matthias Kuhn
* Why is the processing toolbox not an option for your users? (Do you
require shortcuts (menu entries) to the tools? Is it the parameter
interface which is more fine-tuned? Missing functionality?)
* Are the processing toolbox tools enough performant for your needs? If
not, which ones require attention? It would be good to have a list.
If the first one is a problem, I think it would be an approach to let
system administrators define "favorite algorithms" (or similar) which
would then be offered as a shortcut in the menu.
If the second one applies, then we already have proven the point that
maintaining the same algorithm twice is silly and the effort should
rather be put into unifying these tools instead of rewriting duplicate
code. I will probably soon be working on a test suite for processing
which should ensure long-term quality assurance for the algorithms
there. Another advantage of consolidating the two code bases.
I hope that it's possible for your company to consider different
approaches for this problem to get to a long-term enhancement.
Best regards,
Matthias
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20151022/5fca27aa/attachment-0001.sig>
------------------------------
Message: 5
Date: Thu, 22 Oct 2015 12:30:26 +0000
Subject: Re: [Qgis-developer] encoding in custom python functions
Content-Type: text/plain; charset="utf-8"
Ah... I think I now understand what you are saying. You are saying that
expression files are always saved in your system default encoding system.
Sorry for the noise.
Hi Alexandre,
The u'ä' works if I tell the file encoding using your snippet
# -*- coding: utf-8 -*-
But this works only in qgis and is not a correct way.
I cannot then debug my function in any IDE cause the file is encoded
differently than it says.
-Lauri
Have you tried to put the following line in the top of the file?
# -*- coding: utf-8 -*-
Hi all,
I'm trying to figure out some encoding problems when using custom python
expression functions.
I'm using python functions for defining some label texts. I need use
some special characters like ä and ö.
When a new file is created in function editor the file is encoded in my
system encoding cp1252. At least the ä is coded to 0xe4 if I check that
with hex-editor.
Let the custom function just be
@qgsfunction(args='auto', group='Custom')
return 'ä'
This displays on a map like 'ä'
It seems that the file is converted to utf-8 before execution and the ä
is then coded to 0xc3 0xa4. Now QGIS still tries to interpret this as
cp1252 and then displays this as 'ä'.
It doesn't help to define the ä as unicode like u'ä' because QGIS still
sees the ä as 0xc3 0xa4 and think that it is in cp1252.
The only way to resolve this in the code at least that I have found is
to use
return 'ä'.decode('utf-8')
This works on QGIS but the file can't be executed elsewhere (IDE) since
the file is truly in cp1252 and 0xe4 can't be decoded using utf-8.
Does anyone else confirm this and have I understood right the problem?
Should I file a bug report?
Thanks,
-Lauri
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20151022/af0b1804/attachment.html>
------------------------------
Subject: Digest Footer
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
------------------------------
End of Qgis-developer Digest, Vol 120, Issue 96
***********************************************
--
Giovanni Manghi
http://naturalgis.pt/
***@naturalgis.pt
* WebGIS development
* QGIS/PostGIS Training
* QGIS Support and Consulting
* QGIS development
Google+/Hangouts: ***@gmail.com
Giovanni is QGIS main tester and active member of its development team
--
Matthias Kuhn
2015-10-22 13:26:30 UTC
Permalink
Hi Giovanni,
Post by Giovanni Manghi
This users are
usually not able to handle (or willingly to) the possible problems of
using a toolbox based on other software.
There are a number of internal algorithms available in processing (i.e.
no external dependencies).
Unfortunately they are duplicates of ftools algorithms. But they have
the advantage that they can be used in models.
So if something is going to be improved it should be those - or we end
up with a situation where we have fast algorithms in the menus but the
same algorithm slower in the toolbox.

Therefore the plan of action should be:

* Create menu entries for often used (internal) processing algorithms
* Remove ftools algorithms
* Identify slow algorithms
* Improve performance

This will solve João's problems in the most sustainable and effective
way possible.

Chers
Matthias
Mathieu Pellerin
2015-10-22 13:29:05 UTC
Permalink
Yep. I think the user-friendliness of ftools are that a/ algorithms are
offered in a simple and accessible menu items, and b/ the menu item icons
_greatly_ ease the understanding of what algorithms do.

Both can be added to processing.
Post by Matthias Kuhn
Hi Giovanni,
Post by Giovanni Manghi
This users are
usually not able to handle (or willingly to) the possible problems of
using a toolbox based on other software.
There are a number of internal algorithms available in processing (i.e.
no external dependencies).
Unfortunately they are duplicates of ftools algorithms. But they have
the advantage that they can be used in models.
So if something is going to be improved it should be those - or we end
up with a situation where we have fast algorithms in the menus but the
same algorithm slower in the toolbox.
* Create menu entries for often used (internal) processing algorithms
* Remove ftools algorithms
* Identify slow algorithms
* Improve performance
This will solve João's problems in the most sustainable and effective
way possible.
Chers
Matthias
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Matthias Kuhn
2015-10-22 13:35:45 UTC
Permalink
/me adding "icon" to the list of actions

Therefore the plan of action should be:

* Add icon support to processing
* Create menu entries for often used (internal) processing algorithms
* Remove ftools algorithms
* Identify slow algorithms
* Improve performance

Cheers
Matthias
Post by Mathieu Pellerin
Yep. I think the user-friendliness of ftools are that a/ algorithms
are offered in a simple and accessible menu items, and b/ the menu
item icons _greatly_ ease the understanding of what algorithms do.
Both can be added to processing.
Hi Giovanni,
Post by Giovanni Manghi
This users are
usually not able to handle (or willingly to) the possible
problems of
Post by Giovanni Manghi
using a toolbox based on other software.
There are a number of internal algorithms available in processing (i.e.
no external dependencies).
Unfortunately they are duplicates of ftools algorithms. But they have
the advantage that they can be used in models.
So if something is going to be improved it should be those - or we end
up with a situation where we have fast algorithms in the menus but the
same algorithm slower in the toolbox.
* Create menu entries for often used (internal) processing algorithms
* Remove ftools algorithms
* Identify slow algorithms
* Improve performance
This will solve João's problems in the most sustainable and effective
way possible.
Chers
Matthias
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Victor Olaya
2015-10-22 13:59:27 UTC
Permalink
Post by Mathieu Pellerin
Yep. I think the user-friendliness of ftools are that a/ algorithms
are offered in a simple and accessible menu items, and b/ the menu
item icons _greatly_ ease the understanding of what algorithms do.
Icon support is already in Processing. The first versions actually
used the same ftools icons, but I removed that because there were new
algorithms being added, and it was quite messy to see some algorithms
with icons and others with the default icon. Instead, I started using
the same icon for all algorithms for a given provider, for the sake of
homogeneity.

However, the mechanism that I implemented for putting processing
algorithms in a menu allows to set an icon, so the current ftools menu
can be perfectly replicated, including icons

I copy here an example that I added in another thread, about how to
replicate the Raster/Conversion menu from Processing


import os
from GdalTools import GdalTools
from processing.gui.utils import addAlgorithmEntry
from PyQt4.QtGui import *

def addRasterAlgorithm(name, group, iconName):
icon = QIcon(os.path.join(os.path.dirname(GdalTools.__file__),
"icons", iconName + ".png"))
addAlgorithmEntry(name, "Raster", group, icon = icon)

addRasterAlgorithm("gdalogr:rasterize", "Conversion", "rasterize")
addRasterAlgorithm("gdalogr:translate", "Conversion", "translate")
addRasterAlgorithm("gdalogr:pcttorgb", "Conversion", "24-to-8-bits")
addRasterAlgorithm("gdalogr:rgbtopct", "Conversion", "8-to-24-bits")
addRasterAlgorithm("gdalogr:polygonize", "Conversion", "polygonize")

The addAlgorithmEntry is a part of Processing since quite a few
commits ago, and it allows to put an algorithm "outside" of the
toolbox, in a menu entry.

So the code is basically there, it is just a matter of finding out
which algorithms are missing and deciding where to call this methods
from. Then I guess we could use that funding to improve algorithms and
make them more robust and fast


Cheers
Post by Mathieu Pellerin
Both can be added to processing.
Hi Giovanni,
Post by Giovanni Manghi
This users are
usually not able to handle (or willingly to) the possible
problems of
Post by Giovanni Manghi
using a toolbox based on other software.
There are a number of internal algorithms available in processing (i.e.
no external dependencies).
Unfortunately they are duplicates of ftools algorithms. But they have
the advantage that they can be used in models.
So if something is going to be improved it should be those - or we end
up with a situation where we have fast algorithms in the menus but the
same algorithm slower in the toolbox.
* Create menu entries for often used (internal) processing algorithms
* Remove ftools algorithms
* Identify slow algorithms
* Improve performance
This will solve João's problems in the most sustainable and effective
way possible.
Chers
Matthias
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Alexander Bruy
2015-10-22 13:28:22 UTC
Permalink
Some time ago Carson ported at least some fTools
algorithms to QGIS analysis library. Maybe we can
revise this code, update it and then reuse in Processing
and 3rd party plugins?

Core implementation should be faster than Python code,
also maybe we can adopt multithreading approach for
some alforothms.
Post by João Gaspar
Dear QGIS Developers,
I'm writing on behalf of my employer, a company that has mixed,
proprietary and open source (especially QGIS of course) gis
infrastructure, with qgis being adopted each day a little more.
Actually in our migration path we have found a road blocker: the very
bad performances (and sometimes results) of most of geoprocessing
tools in the vector menu. Yes I know that there are alternatives in
the Processing toolbox, but that is not an option for most of our very
basic users: they need to do simple operations (clips, unions,
intersections, etc.) with usually very large/detailed vectors (usually
shapefiles). In this scenario QGIS usually chokes or is at best very
slow (compared to other gis packages, the ones we are trying to get
rid of).
What can be done to improve this tools? Rewriting the tools to improve
efficiency and accuracy?
We available some funds: this is the first time I convince people
above me to put some money in QGIS development, and a decision would
need to me made until the end of November tops, otherwise money will
be used otherwise.
Please leave advice and feedback.
Regards
João Gaspar
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
--
Alexander Bruy
Giovanni Manghi
2015-10-22 14:21:05 UTC
Permalink
Post by Victor Olaya
Icon support is already in Processing. The first versions actually
used the same ftools icons, but I removed that because there were new
algorithms being added, and it was quite messy to see some algorithms
with icons and others with the default icon. Instead, I started using
the same icon for all algorithms for a given provider, for the sake of
homogeneity.
the main issue here is not if leave th vector menu or not, i think we
all agree we should get rid of it and have all the tools in
processing.

The issue here is about the lousy performances (and even wrong
results) of the native qgis vector geoprocessing tools. I think we all
agree that qgis should have a few (at least the most commonly used)
ones, without depending on any 3rd party software.

cheers!

-- G --
Victor Olaya
2015-10-22 14:27:00 UTC
Permalink
Post by Giovanni Manghi
The issue here is about the lousy performances (and even wrong
results) of the native qgis vector geoprocessing tools. I think we all
agree that qgis should have a few (at least the most commonly used)
ones, without depending on any 3rd party software.
I fully agree on that

Do we have any benchmark or list of those algorithms with very bad
performance or that give wrong results? I don't think that all of them
behave so badly, so it would be good to have more detail about where
we should be focusing our efforts in this

Cheers
Giovanni Manghi
2015-10-22 15:07:32 UTC
Permalink
Post by Victor Olaya
Post by Giovanni Manghi
The issue here is about the lousy performances (and even wrong
results) of the native qgis vector geoprocessing tools. I think we all
agree that qgis should have a few (at least the most commonly used)
ones, without depending on any 3rd party software.
I fully agree on that
Do we have any benchmark or list of those algorithms with very bad
performance or that give wrong results? I don't think that all of them
behave so badly, so it would be good to have more detail about where
we should be focusing our efforts in this
great, so I would suggest try underestand why the tools here

https://github.com/qgis/QGIS/blob/master/python/plugins/fTools/tools/doGeoprocessing.py

are so bad with such "problematic" datasets (that I can easily
provide). If is possible to do something
then take João company offer use it to improve the situation, then we
can decide with calm were to put the
shortcuts :)

cheers!

-- G --
Matthias Kuhn
2015-10-22 15:15:41 UTC
Permalink
Giovanni,

Would (some of) these datasets be suitable to include in unit tests for
processing?
It would be very interesting to have these available for this purpose.

Matthias
Post by Giovanni Manghi
Post by Victor Olaya
Post by Giovanni Manghi
The issue here is about the lousy performances (and even wrong
results) of the native qgis vector geoprocessing tools. I think we all
agree that qgis should have a few (at least the most commonly used)
ones, without depending on any 3rd party software.
I fully agree on that
Do we have any benchmark or list of those algorithms with very bad
performance or that give wrong results? I don't think that all of them
behave so badly, so it would be good to have more detail about where
we should be focusing our efforts in this
great, so I would suggest try underestand why the tools here
https://github.com/qgis/QGIS/blob/master/python/plugins/fTools/tools/doGeoprocessing.py
are so bad with such "problematic" datasets (that I can easily
provide). If is possible to do something
then take João company offer use it to improve the situation, then we
can decide with calm were to put the
shortcuts :)
cheers!
-- G --
_______________________________________________
Qgis-developer mailing list
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Continue reading on narkive:
Loading...