CS858 F19 Paper #20 Reviews and Comments
===========================================================================
Paper #20 Property Inference Attacks on Fully Connected Neural Networks
using Permutation Invariant Representations


Review #20A
===========================================================================
* Reviewer: Thomas Humphries <t3humphr@uwaterloo.ca>
* Updated: 19 Oct 2019 4:52:38pm EDT

Paper summary
-------------
The authors explore property inference attacks on neural networks by extending the shadow training technique. Initially they extend the shadow training technique using a vector of all neural net parameters as input to the meta classifier and find this doesn’t perform well. They suggest this is due to the many possible permutations of the vector representation and propose two methods as solutions. The first method sorts the neurons by magnitude of the sum of the weights, the second approach uses a meta classifier that follows the deep sets architecture. Both methods are evaluated, and the deep sets approach is shown to be the most effective with 85 -100 percent accuracy.

Strengths
---------
The deep sets architecture is well suited to the problem and is shown to be quite effective. The extraction of node’s features combined by layer and finally the entire network reduces the possible feature space and is invariant to permutations by it’s construction without any additional overhead.

The approach is described using very clear and rigours notations. They always explain any notations very thoroughly before using.

A good variety of datasets are considered in multiple domains.

Weaknesses
----------
There are huge assumptions of data and computational resources available to the adversary and very little attempt to reduce either of these.

Some of the properties that are tested are very subjective and no information is given as to how they are evaluated. Properties such as wearing glasses would be easy to determine in an experiment but things like “older faces” and “attractiveness” are very vague criteria.

No information is given as to how the datasets are divided into the various training and test sets needed for all the models. Information such as the size of each dataset and if they overlap, especially between training and test sets, is needed to justify these results.

Very little testing is carried out on defenses. This section briefly touches on three different defences with little to no empirical results. The authors mention black box attacks are out of scope of this work however black box attacks may circumvent most of the defences discussed.

The case study sections seem like they are just fill. They mostly just re-iterate results of previous sections.

Opportunities for future research
---------------------------------
The problem of hiding certain properties of a dataset from machine learning algorithms is very interesting and needs more exploration than what is given here. I think it would be interesting to look at developing some kind of property level differential privacy for neural networks which could defend against the attacks presented in this paper along with other approaches.



Review #20B
===========================================================================
* Reviewer: Sung-Shine Lee <s469lee@uwaterloo.ca>
* Updated: 20 Oct 2019 1:05:24pm EDT

Paper summary
-------------
The paper propose two attacks to a model to infer its global property of the underlying dataset on Fully Connected Neural Network. The two attacks are based on the same idea: nodes can be arbitrarily permuted in a hidden layer of a FCNN. Consequently, for the first attack, the authors proposed to sort the nodes first to produce a canonical form. for the second attack, as layers in neural networks are usually represented with vectors, with the permutation equivalence property it can actually use set representation instead of vector representation.

Problems in Understanding
-------------------------
- It is unclear to me why the set based can outperform sorting based method.

Strengths
---------
- The paper first tried previous property inference attacks and found out that they doesn't work well on FCNN, then show that their method works. This gives a nice support of the validity and usefulness of the proposed attack.
- They empirically showed the node permutation with Figure 3, which I find it to be interesting.
- The difference between the set based method and sorting based method should be discussed more thoroughly to understand the underlying mechanism.

Weaknesses
----------
- The method is very specific and can only work on Fully Connected Neural Network.
- The traditional property inference attack was deemed failure after they failed to infer the property: "whether the classifier was trained on images with noise". However it is unclear whether this property can be inferred on models other than FCNN. I would imagine that the effectiveness of property inference is also closely related to the property itself.
- The target model trained has only 3 hidden layers with sizes of 32, 16, and 8. This is small compared to other models.

Opportunities for future research
---------------------------------
- Explore what kinds of property are easier to infer by property inference attacks. This is related to the Weakness mentioned above: "However it is unclear whether this property can be inferred on models other than FCNN"
- What would the performance be if the attack is launched on a model size that is larger and more realistic?



Review #20C
===========================================================================
* Reviewer: Rasoul Akhavan Mahdavi <r5akhava@uwaterloo.ca>
* Updated: 20 Oct 2019 6:52:17pm EDT

Paper summary
-------------
With the widespread use of deep learning models for machine learning purposes, many attacks have been developed to exploit numerous weaknesses in terms of privacy and security. This paper focuses on property inference, following up on work from Ateniese et. al.
Previous work proposed training a meta classifier trained on networks, with and without a specific property. This paper proposed a better representation of neural networks to capture the invariance property of different layers, that can lead to a better performance of the metaclassifier especially with less data points.
Evaluation are conducted with 2 methods with results showing improvement in terms of the number of needed data points and accuracy on 3 different datasets supporting the claims made in the previous section.

Strengths
---------
- Well picked, intuitional solution to the problem of architectures that are invariant to permutations. an architecture like DeepSets is specifically designed for such a scenario.
- Some flaws and alternative solutions are pointed out, especially in the last section, despite the fact that they didn't try many of the proposed future work themselves.

Weaknesses
----------
- While the paper does address a problem in the previous approach from Ateniese et. al, which is the low accuracy, it doesn't overcome the major problem of having to train shadow models in the first place. Training models might just not be feasible, especially when the main study is one neural networks which are known to be hard to train.
- The problem (property inference) is good but the attack doesn't correspond to any realistic scenario.
- The task that MNIST was evaluated on is very vague. Whether an image is noisy or not isn't a well defined property and it certainly can't be distinguished by a human. We might not add noise to an image but might have some inherent noise in it or vice versa. Maybe it would have been better to at least define a mathematical metric to define if an image has noise or not.
- Attacks are analyzed on completely defenseless models, and the defenses that are discussed seem to have incomplete, cherry-picked results. The result of each defense is only stated on one dataset, making it ungeneralizable to different datasets without further analysis.

Opportunities for future research
---------------------------------
- Is there any way to circumvent the major obstacle of having to train shadow models? It seems like too much of a burden in many cases.



Review #20D
===========================================================================
* Reviewer: Laura Graves <laura.graves@uwaterloo.ca>
* Updated: 20 Oct 2019 9:31:35pm EDT

Paper summary
-------------
Property inference attacks that divulge unintended information about the dataset a model was trained on have been shown to be effective against many types of ML models, but fairly ineffective against neural networks. In this paper the authors show that the extremely large number of neural networks that are simply permutations of the same function is part of the problem - there's simply not enough data (nor would there ever be) for a metaclassifier to learn a permutation invariant representation.

To overcome this difficulty, the authors present two novel methods for training a metaclassifier. In the first, they develop a neuron sorting method that will sort the neurons in each layer so that two semantically identical networks with different representations will end up the same. Secondly, they instead use a specific neural network model that's made to deal with sets (the DeepSets architecture), and treat each layer of the network as an unordered set.

Through testing the authors show that both these methods show a significant performance increase over the naive representation.

Strengths
---------
The performance increase is huge! I was very impressed with the author's ability to take a difficult problem and present two novel methods for solving it.

The hardware performance counter problem is really fascinating and a vivid example of unintended attribute leakage. Spectre/heartbleed vulnerability had no part whatsoever in the feature set or any part of the training process and they were still able to be significantly distinguished in the resultant model. This is a good illustration of just how vulnerable these models can be, even totally unintentionally.

Third, the authors specifically point out that these attacks seem to work against non-overfitted models. The models they use seem to generalize perfectly fine, which means these could potentially work against even strongly regularized models.

Weaknesses
----------
First, simple fully connected networks aren't too commonly used in practice, and I'd be surprised to see them used in a privacy-sensitive environment. The results they have are very impressive, but if the approach can't scale to larger models or be adapted to any more sophisticated architectures then it's difficult to see the real-life applicability.

Second, defense methods were discussed but not analyzed. These results are excellent on these simple, undefended models - how much would it take to defend these models?

Opportunities for future research
---------------------------------
I would analyze the relation between regularization techniques and vulnerability to inference attacks. The authors state that their models generalize well, but I suspect that some methods like distillation may give models that aren't as vulnerable, while others like dropout may increase the information learned at each node and unintentionally lead to further leakage.



Review #20E
===========================================================================
* Reviewer: John Abraham Premkumar <jpremkum@uwaterloo.ca>
* Updated: 20 Oct 2019 9:36:44pm EDT

Paper summary
-------------
In this paper, Ganju et al. want to extend effective property inference attacks to the domain of fully connected neural networks (where hitherto attacks were demonstrated against HMMs and SVMs). They assume a white box model for their attack.

They propose an attack that uses shadow classifiers and meta classifiers, but this does not get good accuracy. They hypothesise that this is due to the nature of FCNNs where they are invariant under permutation of nodes in a layer. To address this, they introduce two techniques, one using neuron sorting, and the other using deepnets w/ set based representations rather than vector based.

They proceed to evaluate both of these against their first attack at the baseline, and see that the set based method performs the best. They go on to conduct experiments on inferring vulnerabilities in systems and also inferring training data distributions from attacking the models.

They discuss alternative approaches as well, in brief (as opposed to the two they used here), as well as discuss possible future work and defenses. They conclude by summarizing their work, and in a final note they mention that  permutation invariance of FCNNs could have uses that go beyond just property inference.

Problems in Understanding
-------------------------
Not very sure how the number of parameters for the set based approach is lesser than for the other approaches

Strengths
---------
1: They propose two permutation invariant methods to address the fact that FCNNs are invariant when permuting nodes in each layer.  They experiment to find out the effectiveness of both.

2: Their neuron canonical sorting method (in theory) can be used  with generic machine learning algorithms, though it turns out that the method is not as effective as the one using deepsets.

3: They use more than just image classification tasks in their experiments, and their experiments are extensive.

Weaknesses
----------
1:There is no section for threat model/ adversary model with adversary capabilities etc

2: They assume white box access to the target

3: They focus only on binary class property inference and not multiclass

4: On MNIST and HPC, even the set based meta classifier reaches only around 85% accuracy, even though it reaches more than 90 on others. This is not adequately addressed.

Opportunities for future research
---------------------------------
They have a section for future work, and I am not able to come up with other substantial ideas



Review #20F
===========================================================================
* Reviewer: Lucas Napoleão Coelho <lcoelho@uwaterloo.ca>
* Updated: 20 Oct 2019 9:36:54pm EDT

Paper summary
-------------
The authors adapt the existing property inference attack, where an attacker infers a general property of the training dataset, to fully connected DNN models. The attacks were previously shown to be successful against Hidden Markov Models and SVM classifiers. They identify the main problem of DNN to be the presence of equivalent representations of the same network, based on this observation, they propose a canonical representation of flat DNN weights and biases and a conversion to DeepSet networks, which represent the nodes of each layer as a set, and has orders of magnitude fewer parameters.

Strengths
---------
The authors manage to spot a great way to combine existing research, namely the property inference attacks and the DeepSet networks.

The authors use datasets from different domains and provide excellent motivation for the attack with the HPCs dataset.

The explanations of the paper are very intuitive and easy to follow.

Weaknesses
----------
The writing of the paper is repetitive

It does not explore alternative approaches and possible defences in depth. Which should be, arguably, critical sections of the paper.

They bring up a lot of future work that could easily be included in a more comprehensive paper.

While this is not clearly stated in the paper, they seem to use the same dataset to train the target and the shadow models. In some cases, they say how an attacker could obtain this data in practice, but they should evaluate how important is this match to achieve their results.

Opportunities for future research
---------------------------------
If we set aside the already mentioned future work of extending the work to different types of NN and multi-class tasks, there are a few other lines of future work worth mentioning.

Although challenging, it would be interesting to devise a theoretical bound like differential privacy for property inference attacks. Perhaps, as a first approach, one could try to formalize the information leakage from imbalances in the attributes.

The defences mentioned are either only partially effective or still unexplored at the time of publication of this paper. One could try to apply other defence strategies successful against membership inference, such as ensemble methods.



Review #20G
===========================================================================
* Reviewer: Nils Hendrik Lukas <nlukas@uwaterloo.ca>
* Updated: 20 Oct 2019 10:25:34pm EDT

Paper summary
-------------
The paper explores a property inference attack in a white-box attack setting using only the target network and the ability to produce data from the same distribution to infer global, unintentionally shared properties with close to perfect accuracy. The main contribution is the usage of a permutation invariant projection function of the networks parameters for the feature extraction, which is used to train a meta-classifier. Thereby, DeepSets outperforms all other approaches which is verified on four datasets, one of which has a strong use-case of checking whether training data was gathered from a machine vulnerable to Spectre and Meltdown with obvious security implications.

Strengths
---------
+ Extremely strong and relevant use-case for property inference. The Meltdown and Spectre attack property is actually relevant and represent interesting properties for an adversary.

+ Mentioning of alternate approaches that did not work

+ Discussion of limitations and future work (which was a lot)

+ Good and clear visualizations of their approach

Weaknesses
----------
- Non-Practical attacks with high likelihood. Even if that property can be inferred from the classifier, an attack is still very unlikely as the adversary would have to find the computer on which the model was trained.

- No clear explanation of the data separation method. The authors can produce basically arbitrary data from the same distribution. It is questionable how similar the datasets are and what influencing factors can be identified (CPU-type, OS version ,... )

- No evaluation of defenses

- No clear definition of interestingness of properties

- No evaluation when architectures are different (this is mentioned in future work and I get this is a PoC, nontheless it limits practicability a lot)

Opportunities for future research
---------------------------------
The authors practically mentioned a lot of future research already. I would be highly interested in how property inference performs on a model extracted through black-box.



Review #20H
===========================================================================
* Reviewer: Andre Kassis <akassis@uwaterloo.ca>
* Updated: 20 Oct 2019 11:16:57pm EDT

Paper summary
-------------
The paper studies the possibility of property inference attacks against fully connected neural networks (FCNNs) in the white-box settings. As the sharing of machine learning models is becoming popular, attackers are more likely to gain access to the architecture and weights of publicly available models. The paper hence examines how attackers with such knowledge may infer properties of the training data or the platform on which the model was trained that the model producer did not intend to share.

While previous works in the field demonstrated successful property inference attacks against other ML methods, they were not able to achieve significant success rates when evaluated against FCNNs. The authors attribute this failure to the complexity of FCNNs and the fact that they are invariant under permutations of nodes in the same layers. To launch a property inference attack, shadow models are trained with and without the property, and their weights are used as training inputs for a meta classifier that classifies ML models based on the presence or the absence of the property in the training set. The invariance of FCNNs under permutations makes the number of FCNNs that learn the exact same function considerable and therefore training meta classifiers based on a limited number of shadow models is doomed to fail as these meta classifiers will not be able to represent all the different permutations of FCNNs that perform the same task.

To address this issue, the authors propose two different methods that capture the invariance and simplify the information extraction tasks: 1) sorting the nodes in each layer according to a specific metric and 2) using functions that act on sets to compute features of the different layers rather than functions that operate on vectors.

The authors evaluate their strategies on numerous datasets and different classifiers from different domains with different properties. The experiments show that the proposed methods achieve extremely high attack success rates.

Finally, the authors browse different possible defenses and point out limitations in their attack schemes together with ideas for future research. The authors believe that the permutation invariance property may have other applications beyond property inference.

Problems in Understanding
-------------------------
The paper is very well-written and easy to understand.

Strengths
---------
 - Clever observations and algorithms – The authors make several key observations that explain the ineffectiveness of property inference attacks against FCNNs that stems from the nature of such ML models. Based on these insights, they manage to devise novel, highly effective attack methods that defeat these limitations and achieve high success rates.

 - Extensive experiments on real datasets from different domains – The authors manage to show that their attack method is practical and domain-agnostic by conducting a large number of tests on models that solve various tasks. Furthermore, they show that the attack manages to reveal properties of the training sets and platforms that have very little to do with the main functions of the target models.

 - Discusses future work and limitations – The authors of the papers discuss the limitations of their attack strategy, such as the need for large datasets to train shadow models and propose many ideas for future research.

Weaknesses
----------
 - Assumes the adversary can find large datasets to train the shadow models and the meta classifier – This assumption may be highly unrealistic as it assumes extremely powerful adversaries. Furthermore, the authors do not explain how the data was split between the different models used to train and test the meta classifier, which makes us question the integrity of the paper. It is possible that the attack was only successful because all the models were highly similar as they were trained on the same data.

 - No explanation for the reasons behind choosing specific proportions of samples with/without the properties in the training sets – In their experiments, the authors set different ratios for examples with a particular property in the training set of the models and based on these ratios they determine whether the model belongs to P or P`. However, these ratios differ significantly from one classification task to another without any explanations. It is reasonable to assume that these ratios worked the best for the authors, and therefore they reported them and them only. Hence, more details are required, and exploring other rates is vital for determining the true capabilities of the attacks.

 - Weak case study and motivation behind the attack – The authors claim that their attack is highly useful when an adversary wants to infer vulnerabilities in the platform on which the target model was trained. However, the attack requires enormous resources, and the attacker’s gain appears to be insignificant in this case. It is much easier for the attacker (and less time-consuming) to try and exploit the vulnerability in the target system instead of having to find massive datasets and train several models only to determine whether the vulnerability is present or not.

Opportunities for future research
---------------------------------
 - Examine the use of methods that obfuscate the final learned weights of the FCNN model – The attack relies on the resemblance between the weights of the models trained with and without a specific property. Hence, a possible defense would be to try and eliminate such similarities. For instance, the model owner can use locality preserving hashing techniques to hash the inputs before feeding them to the model. This strategy will result in the NN learning different weights depending on the hash function, and without knowing the function, the attacker will not be able to place the attack.

 - Enhance the performance of attacks that sort the nodes in each layer – Functions acting on sets must treat the different elements in the sets in the same way, which might result in losing valuable information leading to a weaker attack. However, the other method proposed in the paper that sorts the nodes in each of the layers does not exhibit such a weakness. That said, the authors claim that the former method had better results in their experiments and attribute this outcome to the fact that it is harder to train meta classifiers using the latter approach due to a large number of parameters needed. A promising approach would be to perform a certain reduction operation on the output of the sorting algorithms and then feed the output to the meat-classifier to extract the largest amount of information possible.



Review #20I
===========================================================================
* Reviewer: Karthik Ramesh <k6ramesh@uwaterloo.ca>
* Updated: 20 Oct 2019 11:21:15pm EDT

Paper summary
-------------
This paper attempts to build a property inference attack against fully connected neural networks. They aim to infer properties that are global to the training dataset, rather than individual records like membership inference attacks from previous research has done. They first try to build shadow models that contain the property they are trying to infer and build a classifier that detects their presence/absence. This approach produces accuracy close to random and they hypothesize that this is due to the “permutation” property of the neural network – which leads multiple neural networks using permutations of a set of nodes based on a base neural network to produce the same result as the base network. And since the network representation they used earlier to train the classifier does not account for this permutation property, they provide enhancements that account for this. They introduce a set-based and sorting-based representation to account for this. Their evaluations show that this counters the permutation property of the neural network and enables the property inference attacks. They also try to evaluate a few enhancements/defences for this technique.

Problems in Understanding
-------------------------
1.	What is the training process for the \phi in the set-based representation? What is the ground-truth used to build these networks?

Strengths
---------
1.	This paper evaluates their technique on HPCs that could lead to a system attack rather than just a model privacy loss
2.	I liked that they talked about graph-based representation and experimented with it
3.	Very well written for the most part with clear representations and explanations.

Weaknesses
----------
1.	When an attacker has sufficient data to conduct this attack on a dataset like MNIST, I don’t think the attacker would have any motive to do so.
2.	No mention of how many exact data points was used to train the set-based shadow models whose representation is then learnt by the meta-classifier. Only the number of shadow models is mentioned.
3.	No mention of the training of the \phi network for the set-based representation.

Opportunities for future research
---------------------------------
1.	This technique could probably be extended to verify (by a third-party enforcer before being deployed) whether the model has been trained with bias on any property of the data. This can even be used to verify models that are actively learning to check that they are neutral (referring to Microsoft's Tay).



Review #20J
===========================================================================
* Reviewer: Matthew David Rafuse <mrafuse@uwaterloo.ca>
* Updated: 20 Oct 2019 11:47:47pm EDT

Paper summary
-------------
In this paper, Ganju et al. investigate property inference attacks on Fully Connected Neural Networks. In the past, due to the relative complexity of FCNNs, there has been limited success in property inference attacks against them. The authers propose a new method, using the fact that FCNNs are invariant under permutation to significantly lower the complexity of the attack. This simplifies the attack and increases it's efficacy on complex models. Their attack is evaluated on real world examples, including some interesting case studies.

Problems in Understanding
-------------------------
I was a bit confused as to whether permutation invariant representations could only be used with FCNNs or any NN, regardless of whether they are fulling connected or not. How would other layer types be represented in this format?

Strengths
---------
1. The reduction in complexity (measured by number of parameters) is truly impressive. We see orders of magnitude fewer parameters necessary using the set-based method.

2. The theory around Permutation Invariant Representations is well done, and made for an interesting read.

3. The Spectre/Meltdown case study was really cool, albeit without much practical purpose or it being a realistic attack, in reality. in fact, both case studies were interesting in themselves, outside of demonstrating their implementations.

Weaknesses
----------
1. This paper was pretty tough to read. The font was really small and some parts flowed pretty poorly. The paper was really dense to look at without much visual space. There was also a fair amount of inconsistency in their representation in data - for example, tables 2 and 4 both list percentages, but in different formats - one on the interval $[0, 1]$ and the other on the interval $[0, 100]$.

2. Several assumptions made in Section 7.5 do not hold up to scrutiny. For the detection of spectre patches, it is necessary to train on a dataset identical to the computer used to create the dataset in the first place. They brush this off, but it seems almost impossible to perfectly replicate the environment in any realistic scenario - there are simply too many moving parts.

3. As far as I can tell, their performance dataset was not released, making reproduction of their results a bit more difficult for that particular case study. To their credit, they do explain their methodology for collecting your own data.

Opportunities for future research
---------------------------------
Consider other computational layers for this sort of property inference attack, as mentioned in the paper.



Review #20K
===========================================================================
* Reviewer: Viet Hung Pham <hvpham@uwaterloo.ca>
* Updated: 20 Oct 2019 11:49:51pm EDT

Paper summary
-------------
The paper introduces a property inference attack on Fully-connected neural networks that uses the Permutation Invariant Representations. These representations are extracted on converted canonical models parameters and in sets. By changing the features set representing target models, the paper shows that the proposed variation of shadow training attacks works well on models trained for five ML tasks.

Problems in Understanding
-------------------------
The paper is overall easy to understand.

Strengths
---------
+ Give a good overview of the shadow training attack.

+ The intuition explanation of the problem when applying shadow training inference attack on neural networks. The symmetry nature of the permutation equivalence is interesting.

+ The proposed solutions match well with the observation and intuitively explained.

Weaknesses
----------
- The paper focuses on the fully connected model where the number of permutation equivalence models is large. The paper should try to experiment on CNN as it has fewer parameters which will have fewer equivalence models.

- The experiments with only four datasets are somewhat limited. CNN  might be a better architecture for the image classification tasks (CelebA and MNIST) instead of FCNN.

- It would be interesting to see the attack resources requirement as it seems very expensive (requires over 2500 shadow models to work well in most cases)

- The detailed result for the possible defenses could be added. For example, the detection accuracy for models that use ReLU activation function.

Opportunities for future research
---------------------------------
+ Can we introduce in the original loss function (for the original target model) an additional term that would minimize the effectiveness of a meta-classifier trying to infer certain protected properties? Would these be a way to mitigate inference attacks on these protected properties? So overall, the idea is can we use GAN to build a less leaky network? In this case, the generator would be the training process of the original model. The ground truth would be these shadow models. And the discriminator would be the meta-classifier. The final model output would be the model that comes out of the training process without being labeled as containing protected properties.

+ Can the meta-classifier have adversarial examples? Is there a way to perturb the parameters of the original model such that the meta-classifier would fail? Since adversarial examples transfer between models, maybe the original model owner could do this to prevent the attacker from knowing the protected properties. The perturbation can be designed in such a way that after the canonical conversion the perturbation is amplified (i.e., before conversion the L_inf norm is small, but after conversion the L_inf norm increase)?

+ Would CNN have the same level of high permutation equivalence as fully-connected models? Since the number of parameters of a CNN would be smaller than that of an FCNN, would the original shadow training attack still work?



Review #20L
===========================================================================
* Reviewer: Vineel Nagisetty <vnagisetty@uwaterloo.ca>
* Updated: 20 Oct 2019 11:59:28pm EDT

Paper summary
-------------
In this paper, Ganju et al. leverage the permutation invariance property of fully connected neural networks to design efficient property inference attacks. They use two such permutation invariant ideas: sorting and sets, and show improvement in property inference attacks effectiveness using these ideas. They apply these attacks in different settings. Finally, they give some insights into possible defenses and alternative approaches for attacks.

Strengths
---------
1. The paper is structured well and the end of Section 1 clearly outlines the rest of the paper.

2. The idea of leveraging permutation equivalence and using deep sets is simple and effective. Figure 3 is a great visual of permutations of various neural networks learning on the same data.

3. Using the HPC data to find vulnerabilities in the model that generated bitcoin dataset is novel and practical.

Weaknesses
----------
1. The proportion percentage used in the experiment (Table 1) seems arbitrary. An adversary might need to conduct the same experiment with various proportions to find P.



Review #20M
===========================================================================
* Reviewer: Nivasini Ananthakrishnan <nanantha@uwaterloo.ca>
* Updated: 20 Oct 2019 11:59:55pm EDT

Paper summary
-------------
Property inference attacks done by training shadow models on different training sets have low success rates in practise for deep neural networks including Fully Connected Neural Networks (FCNNs). The paper proposes two ways to feasibly launch successful property inference attacks on FCNNs by utilizing permutation invariance property of FCNNs.

The paper does this by proposing two representations for shadow models. A meta-classifier is built based on these representations to classify each model as trained on set with property or without property. One representation is a fixed ordering of all neurons in a layer. Another ordering is one that is learnt using Deep Sets architecture.

Problems in Understanding
-------------------------
I did not understand why white box access to the target model is required. To build the shadow model models, it seems like we only need to know the architecture of the target model and be able to query the target model.

Strengths
---------
The reason for the complexity of previous property inference attacks on FCNNs is identified. This cause of complexity is mitigated by the mechanisms proposed in the paper. The performance of attacks is much better than the baseline attacks both in terms of efficiency and accuracy.

Weaknesses
----------
A property P holding on a dataset is said to be having over fraction f of dataset having property P. The fraction f for properties in the evaluation seem arbitrary. There are fractions like .59, .37. It seems like these fractions were chosen to have high success rates of the property inference attacks.



Review #20N
===========================================================================
* Reviewer: Tosca Lechner <tlechner@uwaterloo.ca>
* Updated: 21 Oct 2019 12:05:19am EDT

Paper summary
-------------
In this paper the authors propose two strategies of finding equivalences between neural network representations to perform successful property inference attacks.

Problems in Understanding
-------------------------
why does the set-based approach work better than sorting?
I didn't understand the details of their approaches

Strengths
---------
insightful approach: they use a nice structural insight about fully connected layers and how to minimize possibilities

interesting problem: property inference attacks

their approach can be combined with other kinds of attacks (?)

nice visualizations

different domains/tasks were considered

Weaknesses
----------
- limited to only fully connected layers

- white-box access assumed

-paper not publicly available? no source code available?

Opportunities for future research
---------------------------------
There are several future research ideas outlined in the paper: extending to not fully-connected neural networks, extending to multi-classification and to regression, coming up with defenses, extending to membership inference



Review #20O
===========================================================================
* Reviewer: Iman Akbari <iakbariazirani@uwaterloo.ca>
* Updated: 21 Oct 2019 12:05:25am EDT

Paper summary
-------------
The authors of this paper propose a learning approach to white-box property inference attacks in fully-connected feed-forward networks and argue that a permutation-indifferent representation of the network helps the attack perform better. The attack is formulated as a binary classification task which takes the model representation as an input and decides whether a certain property holds for the dataset on which the model was trained. In order to perform the classification, a “meta-dataset” of models trained on different datasets that might or might not have the target property P must be generated.

As an alternative to the flattened weights vector representation, two simple different strategies are proposed: sorting neurons in each layer and a set-representation of weight values. It is shown that using these representations, the trained model performs better than the classic representation which can insinuate that permutation equivalence has an important role in such attacks. Specifically, the set representation out-performs the baseline flat-vector representation significantly. For instance, in detecting a higher percentage of women in the training sample of the US Census income dataset, the set representation achieves 97% accuracy which is almost twice the baseline’s 55%.

Strengths
---------
Frankly, I can't think of a lot of strengths in this one, other than:
- the simplicity of the proposed attack
- an extensive discussion of their limits and future work section

Weaknesses
----------
-	Gathering the “meta-dataset” is computationally expensive
-	The attack seems to be easy to avoid using the classic defence methods we’ve seen so far e.g. dropout, not much evaluation is given in that area
-	The whole method revolves around a representation of FNNs, but it doesn’t generalize for other architectures and this is a significant weakness
-	Only evaluated for simple tasks, probably due to the high-performance cost which might show that the attack is not feasible at scale
-	Lots of inaccurate or over-exaggerated statements like:
	•	“The white-box assumption is reasonable and quite common nowadays” -> just because there are many public models doesn’t mean white-box attack model is de facto
	•	“It has been demonstrated that adversaries can efficiently extract target models with near-perfect fidelity for popular model classes” -> It has been shown that effective model inference attacks do exist in many situations. Stating it as a generally solved problem is putting it lightly.
-	Over explanation of basic concepts such as permutation equivalence

Opportunities for future research
---------------------------------
Trying to find better ways to represent or finger-print a network e.g. by adversarial examples, a representation based on decision boundaries, etc.

Studying the effect of classic defence methods (e.g. perturbation) or the neuron injection  method presented in paper #19 against these attacks.