Parallel calculation for cases with polygons

[richardurs]

Hi,

I'm using Circuitscape.jl on my 4 core PC. Even if I configure max_parallel = 8 the calculation of solving pairs it's always using just one task. It even doesn't seem to be faster than the normal Circuitscape 4.05 Tool. I'm not sure about what I'm doing wrong. My ini File looks like that:

`[Options for advanced mode]
ground_file_is_resistances = True
remove_src_or_gnd = keepall
ground_file = (Browse for a ground point file)
use_unit_currents = False
source_file = (Browse for a current source file)
use_direct_grounds = False

[Mask file]
mask_file = None
use_mask = False

[Calculation options]
low_memory_mode = False
parallelize = True
solver = cholmod
print_timings = False
preemptive_memory_release = False
print_rusages = False
max_parallel = 8
precision = float

[Short circuit regions (aka polygons)]
polygon_file = (Browse for a short-circuit region file)
use_polygons = False

[Options for one-to-all and all-to-one modes]
use_variable_source_strengths = False
variable_source_file = None

[Output options]
set_null_currents_to_nodata = False
set_focal_node_currents_to_zero = False
set_null_voltages_to_nodata = False
compress_grids = False
write_cur_maps = 1
write_volt_maps = False
output_file = S:/Projekte/001 ALG/Ökologische Infrastruktur/Circuitscape/150m/150m.out
write_cum_cur_map_only = False
log_transform_maps = False
write_max_cur_maps = False

[Version]
version = 4.0.5

[Options for reclassification of habitat data]
reclass_file = (Browse for file with reclassification data)
use_reclass_table = False

[Logging Options]
log_level = INFO
log_file = S:/Projekte/001 ALG/Ökologische Infrastruktur/Circuitscape/150m/150m.log
profiler_log_file = None
screenprint_log = False

[Options for pairwise and one-to-all and all-to-one modes]
included_pairs_file = (Browse for a file with pairs to include or exclude)
use_included_pairs = False
point_file = C:/OekologischeInfrastruktur/AnalyseFunktionalitaet/Geoprocessing/metaobjektegildearaster_buffered_150.asc

[Connection scheme for raster habitat data]
connect_using_avg_resistances = False
connect_four_neighbors_only = True

[Habitat raster or graph]
habitat_map_is_resistances = False
habitat_file = C:/OekologischeInfrastruktur/AnalyseFunktionalitaet/Geoprocessing/1576_druchlaessigkeit_gilde_a_150.asc

[Circuitscape mode]
data_type = raster
scenario = pairwise

`

I'm hopefuly looking forward someone can help me.

[ranjanan]

@richardurs thanks for trying the new Circuitscape. Would you mind sharing your example in a small Google drive or Dropbox link so I can take a look at it?

[richardurs]

Hi Ranjan Thank you very much taking time for my question. I put the relevant project data in a RAR-File and attached it to this email. I also added a screen shot of my Task Manager. You can see one task is working hard and there other are quiet. Best regards Urs Von: Ranjan Anantharaman Gesendet: Freitag, 14. September 2018 16:29 An: Circuitscape/Circuitscape.jl Cc: Urs Richard; Mention Betreff: Re: [Circuitscape/Circuitscape.jl] Parallel calculation (#148) @richardurs thanks for trying the new Circuitscape. Would you mind sharing your example in a small Google drive or Dropbox link so I can take a look at it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ranjanan]

@richardurs unfortunately I don't attaching the relevant files via email and replying to Github works. You have to upload them directly to Github by finding this page and dragging and dropping the files onto the comment box.

[richardurs]

here is the link: https://www.dropbox.com/s/vq20bfqjhnzuu1c/CS%20Data.rar?dl=0

[ranjanan]

@richardurs I figured out why it's not parallelising. This happened on the old Circuitscape as well. Since you have polygons in your point map and so the graph is being reconstructed and solved every time. If your point map didn't have polygons it would use all the 8 processes.

[emilio-berti]

Hi @richardurs @ranjanan,
I am having the same issue. I understand the problem, but how can I solve it, given that I have polygons? Is there an easy "trick" that I can use? I thought about putting the resistance of the focal points in the polygon to something close to 0, but then how can I re-compact the output matrix back again (i.e. not accounting for more than 1 point from a polygon)?
Best,
Emilio

P.S. I forgot to thank you for the work put in this, it is the only Circuitscape distribution I could install in my Linux machine. Thanks! :)

[ranjanan]

@emilio-berti unfortunately, I do not know of any such trick. Please do let me know if you find one!
From my perspective, I should just rewrite this to be parallel. However, I have a few other more urgent milestones that i need to hit with this project before I shift my attention to this particular issue.

[emilio-berti]

@ranjanan A possible solution would be to sample a random point from each polygon n and consider only that, setting the resistance of the other points of the polygon very low, then reiterate over a number of time m. Taking m relatively high (100?) this should give the average effective resistance between two nodes. It works in parallel and it requires n x m computations in pairwise mode, which is fast enough for me (1 hour with 2129 focal points and 216447 nodes for each iteration with 8CPUs). Does it seem right to you, or is there an issue I don't see here?

[vlandau]

Wouldn't centroids (points) with Short circuit regions for the polygons be equivalent to using polygons?

[vlandau]

@ranjanan can you speak to whether my guess above is accurate? If it is, maybe we should consider that as the solution and close this issue?