[논문리뷰]PACT: PARAMETERIZED CLIPPING ACTIVATION FOR QUANTIZED NEURAL NETWORK

간단하게 이전에 읽은 논문 정리 아주 간단간단

논문은 링크에서 볼 수 있다.

What?

activation quantization을 accuracy degradation 없이하는 방법에 관한 논문이다.

When?

Training time

How?

activation quantization에 적합한 함수를 학습시점에 학습한다.

reference : PACT: Parameterized Clipping Activation for Quantized Neural Networks
그래프로 표현하면 다음과 같으며 알파는 변수이다.

Why?

Quantization of weights is equivalent to discretizing the hypothesis space of the loss function with respect to the weight variables

reference : PACT: Parameterized Clipping Activation for Quantized Neural Networks

결과

reference : PACT: Parameterized Clipping Activation for Quantized Neural Networks
reference : PACT: Parameterized Clipping Activation for Quantized Neural Networks
reference : PACT: Parameterized Clipping Activation for Quantized Neural Networks
reference : PACT: Parameterized Clipping Activation for Quantized Neural Networks

we achieve ∼14× improvement in density when the bit-precisions of both activations and weights are uniformly reduced from 16 bits to 2 bits

quantizing from 16 to 4 bits leads to a 4× increase in peak FLOPs but a 4.5× improvement in performance.


we see a near-linear increase in performance for up-to 4 bits and a small drop at extreme quantization levels (2 bits)

댓글을 남겨주세요~