3.4 NNs Scenario

[bhack]

I have a question about the NNs scenario in 3.4.
An hypotesis about feeding specifical adversarial examples that could go to saturate the network capacity generating a proliferation of new supermasks tasks or simpler supermasks proliferation by more or less long tail samples in a continous learning setup.
Have you never thought about exploring a solution for supermask expansion e.g. extending the Supermask with more resoruces/connection over the same task?
Have you never explored about assign temporary supermasks and have a "consolidation phase" where new supermasks could be "merged" with previous supermasks or promoted as a new (Independent) task/Mask allocation?

[mitchellnw]

That is a good idea, no we have not thought about this!
It is difficult as supermasks which do the same thing could look very different.

[bhack]

Yes It Is a not so much explored topic.
I found some early step about reasoning on masks simliarity in Bespoke vs. Prêt-à-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding but It Is not strictly what we are talking about here.