[QGIS-Developer] How to deal with time delay in QgsProcessingAlgorithm?

[Nils Nolde]

Hi,

I'd like to extend the openrouteservice API client plugin with a
processing provider, but I run into problems implementing an error catch
for 'Query limit exceeded' which is usually set to 40 requests/min. I
want the plugin to be able to continue requesting when it hits that
limit, after waiting the appropriate amount of time.

Currently I'm doing the following:

- The request is made in a 'client' module. That client module is called
by the processing algorithms (e.g. isochrones) whose input feature
source can have more than 40 features and likely exceed the minutely limit

- When the limit is exceeded, the 'client' module catches the HTTP error
and invokes a time.sleep() for (60 - seconds_since_first_request)
seconds, and then continues requesting

That freezes QGIS obviously, which is not what I want.

Any idea how to deal with this? I have no idea how to delay Python
execution without freezing the main application..

Many thanks

Nils

[Nyall Dawson]

Post by Nils Nolde
Hi,
I'd like to extend the openrouteservice API client plugin with a
processing provider, but I run into problems implementing an error catch
for 'Query limit exceeded' which is usually set to 40 requests/min. I
want the plugin to be able to continue requesting when it hits that
limit, after waiting the appropriate amount of time.
- The request is made in a 'client' module. That client module is called
by the processing algorithms (e.g. isochrones) whose input feature
source can have more than 40 features and likely exceed the minutely limit
- When the limit is exceeded, the 'client' module catches the HTTP error
and invokes a time.sleep() for (60 - seconds_since_first_request)
seconds, and then continues requesting
That freezes QGIS obviously, which is not what I want.
Any idea how to deal with this? I have no idea how to delay Python
execution without freezing the main application..

By default algorithms are executed in a background thread, so I'm
surprised to hear that you're getting a freeze here. Did you change
your algorithm to prevent this background execution? How are you
launching the algorithm?

Nyall

Post by Nils Nolde
Many thanks
Nils
_______________________________________________
QGIS-Developer mailing list
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[Nils Nolde]

Hi Nyall,

Ok, I wasn't sure if they're being executed in another thread by default.

Then it is weird. I'm not doing anything out of the ordinary I guess.
I'm calling the client from within the algo
<https://github.com/nilsnolde/OSMtools/blob/345f9dce7893453baa80e22067fabdd17852e5ea/ORStools/proc/isochrones_proc.py#L164>'s
processAlgorithm() and the client sleeps on exception here
<https://github.com/nilsnolde/OSMtools/blob/345f9dce7893453baa80e22067fabdd17852e5ea/ORStools/core/client.py#L161>.
But even when it's not sleeping and only requesting, it's freezing the app.

The algo from the branch in the links above works (not much else though
atm), in case anyone wants to see it in action. There is an API key in
that branch, so all ready to go. Will delete the key soon of course. If
you try: best on a (single geometry) point layer with > 200 points.

Anyways, hope someone can clear up the mystery.

Thanks
Nils

Post by Nyall Dawson

Post by Nils Nolde
Hi,
I'd like to extend the openrouteservice API client plugin with a
processing provider, but I run into problems implementing an error catch
for 'Query limit exceeded' which is usually set to 40 requests/min. I
want the plugin to be able to continue requesting when it hits that
limit, after waiting the appropriate amount of time.
- The request is made in a 'client' module. That client module is called
by the processing algorithms (e.g. isochrones) whose input feature
source can have more than 40 features and likely exceed the minutely limit
- When the limit is exceeded, the 'client' module catches the HTTP error
and invokes a time.sleep() for (60 - seconds_since_first_request)
seconds, and then continues requesting
That freezes QGIS obviously, which is not what I want.
Any idea how to deal with this? I have no idea how to delay Python
execution without freezing the main application..

By default algorithms are executed in a background thread, so I'm
surprised to hear that you're getting a freeze here. Did you change
your algorithm to prevent this background execution? How are you
launching the algorithm?
Nyall

Post by Nils Nolde
Many thanks
Nils
_______________________________________________
QGIS-Developer mailing list
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[Nils Nolde]

Hi,

somehow it fixed itself.. Now, it's not freezing the app, I'm very
pleased:) I didn't specify the output feature sink before, as I was
still testing. So maybe that made a difference. Anyways, consider the
question closed.

One other thing concerning styling output layers from processing
algorithms, but I'll open another question for that. Actually I'll do it
on Stack Exchange first.

Thanks!

Nils

Post by Nils Nolde
Hi Nyall,
Ok, I wasn't sure if they're being executed in another thread by default.
Then it is weird. I'm not doing anything out of the ordinary I guess.
I'm calling the client from within the algo
<https://github.com/nilsnolde/OSMtools/blob/345f9dce7893453baa80e22067fabdd17852e5ea/ORStools/proc/isochrones_proc.py#L164>'s
processAlgorithm() and the client sleeps on exception here
<https://github.com/nilsnolde/OSMtools/blob/345f9dce7893453baa80e22067fabdd17852e5ea/ORStools/core/client.py#L161>.
But even when it's not sleeping and only requesting, it's freezing the app.
The algo from the branch in the links above works (not much else
though atm), in case anyone wants to see it in action. There is an API
key in that branch, so all ready to go. Will delete the key soon of
course. If you try: best on a (single geometry) point layer with > 200
points.
Anyways, hope someone can clear up the mystery.
Thanks
Nils

Post by Nyall Dawson

Post by Nils Nolde
Hi,
I'd like to extend the openrouteservice API client plugin with a
processing provider, but I run into problems implementing an error catch
for 'Query limit exceeded' which is usually set to 40 requests/min. I
want the plugin to be able to continue requesting when it hits that
limit, after waiting the appropriate amount of time.
- The request is made in a 'client' module. That client module is called
by the processing algorithms (e.g. isochrones) whose input feature
source can have more than 40 features and likely exceed the minutely limit
- When the limit is exceeded, the 'client' module catches the HTTP error
and invokes a time.sleep() for (60 - seconds_since_first_request)
seconds, and then continues requesting
That freezes QGIS obviously, which is not what I want.
Any idea how to deal with this? I have no idea how to delay Python
execution without freezing the main application..

By default algorithms are executed in a background thread, so I'm
surprised to hear that you're getting a freeze here. Did you change
your algorithm to prevent this background execution? How are you
launching the algorithm?
Nyall

Post by Nils Nolde
Many thanks
Nils
_______________________________________________
QGIS-Developer mailing list
List info:https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe:https://lists.osgeo.org/mailman/listinfo/qgis-developer

[Nyall Dawson]

Hi,
somehow it fixed itself.. Now, it's not freezing the app, I'm very pleased:) I didn't specify the output feature sink before, as I was still testing. So maybe that made a difference. Anyways, consider the question closed.
One other thing concerning styling output layers from processing algorithms, but I'll open another question for that. Actually I'll do it on Stack Exchange first.

I'll reply on that stackexchange thread.

Nyall

Thanks!
Nils
Hi Nyall,
Ok, I wasn't sure if they're being executed in another thread by default.
Then it is weird. I'm not doing anything out of the ordinary I guess. I'm calling the client from within the algo's processAlgorithm() and the client sleeps on exception here. But even when it's not sleeping and only requesting, it's freezing the app.
The algo from the branch in the links above works (not much else though atm), in case anyone wants to see it in action. There is an API key in that branch, so all ready to go. Will delete the key soon of course. If you try: best on a (single geometry) point layer with > 200 points.
Anyways, hope someone can clear up the mystery.
Thanks
Nils
Hi,
I'd like to extend the openrouteservice API client plugin with a
processing provider, but I run into problems implementing an error catch
for 'Query limit exceeded' which is usually set to 40 requests/min. I
want the plugin to be able to continue requesting when it hits that
limit, after waiting the appropriate amount of time.
- The request is made in a 'client' module. That client module is called
by the processing algorithms (e.g. isochrones) whose input feature
source can have more than 40 features and likely exceed the minutely limit
- When the limit is exceeded, the 'client' module catches the HTTP error
and invokes a time.sleep() for (60 - seconds_since_first_request)
seconds, and then continues requesting
That freezes QGIS obviously, which is not what I want.
Any idea how to deal with this? I have no idea how to delay Python
execution without freezing the main application..
By default algorithms are executed in a background thread, so I'm
surprised to hear that you're getting a freeze here. Did you change
your algorithm to prevent this background execution? How are you
launching the algorithm?
Nyall
Many thanks
Nils
_______________________________________________
QGIS-Developer mailing list
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[Nils Nolde]

Did you change your algorithm to prevent this background execution? How are you
launching the algorithm?

Sorry, didn't really answer this. I'm launching it from the Processing
Toolbox. Didn't try a Model or script yet.

Cheers

Nils