Question: Source identification on dark background?

[keflavich]

I've noticed sources being missed when they are on a relatively dark background. I'm trying to extract sources toward a dark cloud; the background is bright and structured throughout most of the field. Here's a small example cutout:

Top-left and bottom-right images are both the same - they are the original data. Bottom right has x's marking where crowdsource found sources. Top-right is model+sky. Bottom-left is data-(model+sky).

There are clearly several sources missed in the dark lane especially. By contrast, in part of the field with a flatter background, it looks like most or all sources were recovered. If anything, too many sources are ID'd - some of those toward the top-center look questionable:

Any tips on what I can tune to improve the source recovery in the dark lanes?

[schlafly]

Re questionable IDs---crowdsource tries to find sources down to five sigma. If you don't trust the faint end, that suggests you either need to boost the threshold or increase your estimate of the uncertainties on each pixel.

I note thet PSF doesn't look terribly good to me? I've forgotten if these images are undersampled or if you're allowing the PSF to be fit. But it looks like the PSF being used is too sharp leading to all the negative residuals at the centers.

Re dark lanes... there's not a great approach here native to crowdsource. You could change the default npix=20 argument here:
https://github.com/schlafly/crowdsource/blob/master/crowdsource/crowdsource_base.py#L761
to get a finer sky scale. That will give the model more flexibility to absorb the small scale sky variations. But it's just zeroing out the median of the image iteratively after subtracting stars on that scale, and as you approach the PSF FWHM it will zero your entire image and start doing more damage to the PSF. I don't have a better suggestion there.

Once one has sources, @andrew-saydjari 's CloudCovErr is exactly what you want for improving the photometry in regions of rapidly varying background.

[keflavich]

Thanks. The problem is sources that should be detected but aren't; perhaps I need to decrease the uncertainties. But I think the dark lane is the problem: the peaks are (probably) locally >5sigma, but globally below. I'll see about playing with npix; it looks like I need a filter closer to ~10-15 pix on a side to capture this kind of structure in these images.

The PSF is indeed imperfect, and I'm not refitting it right now.

Thanks for those suggestions, it gives me a lot to work with.

[schlafly]

Yes, sorry, I addressed your points in the reverse order---this comment "if anything, too many sources are ID'd - some of those toward the top-center look questionable" would motivate boosting uncertainties. But the sources in the flat part of the image look okay to me---it's not obvious to me that the uncertainties are problematic.

As you say, not resolving the dark lane motivates decreasing the sky filtering scale.

[andrew-saydjari]

I am going to hop on here to comment that @keflavich only have done one pass in the past might be the core explanation for this problem.