close_btn
?

단축키

Prev이전 문서

Next다음 문서

+ - Up Down Comment Print
?

단축키

Prev이전 문서

Next다음 문서

+ - Up Down Comment Print

1.  Neurons Layers 갯수를 늘리기

 

# 일단 모델을 더욱 깊게 만들어서 전체적인 성능을 높여보겠다

 

# Cell # 14

n_inputs = 28*28  # MNIST

n_hidden1 = 400

n_hidden2 = 300

n_hidden3 = 300

n_hidden4 = 300

n_outputs = 10

# 레이어를 늘리고, w 100~200에서 300~400개로 늘렸다.

 

# Cell # 15

reset_graph()

X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")

y = tf.placeholder(tf.int64, shape=(None), name="y")

 

# Cell # 16

# 깊어질수록 Vanishing Gradient 문제가 심해진다. 이를 해결하기 위해 함수를 시그모이드에서 렐루로 바꿨다.

# 또한 렐루 함수를 위해 he_init으로 weight initialization 해줬다.

he_init = tf.contrib.layers.variance_scaling_initializer()

with tf.name_scope("dnn"):

    hidden1 = tf.layers.dense(X, n_hidden1, name="hidden1",  kernel_initializer= he_init,

                             activation=tf.nn.relu)

    hidden2 = tf.layers.dense(hidden1, n_hidden2, name="hidden2", kernel_initializer= he_init,

                             activation=tf.nn.relu)

    hidden3 = tf.layers.dense(hidden2, n_hidden3, name="hidden3", kernel_initializer= he_init,

                             activation=tf.nn.relu)

    hidden4 = tf.layers.dense(hidden2, n_hidden4, name="hidden4", kernel_initializer= he_init,

                             activation=tf.nn.relu)

    logits = tf.layers.dense(hidden4, n_outputs, name="outputs")

 

# Cell # 17

with tf.name_scope("loss"):

    xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)

    loss = tf.reduce_mean(xentropy, name="loss")

 

# Cell # 18

learning_rate = 0.01

with tf.name_scope("train"):

    optimizer = tf.train.GradientDescentOptimizer(learning_rate)

    training_op = optimizer.minimize(loss)

 

# Cell # 19

with tf.name_scope("eval"):

    correct = tf.nn.in_top_k(logits, y, 1)

    accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))

 

# Cell # 21

init = tf.global_variables_initializer()

saver = tf.train.Saver()

 

# Cell # 22

n_epochs = 20

n_batches = 50

 

with tf.Session() as sess:

    init.run()

    for epoch in range(n_epochs):

        for iteration in range(mnist.train.num_examples // batch_size):

            X_batch, y_batch = mnist.train.next_batch(batch_size)

            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})

        acc_train = accuracy.eval(feed_dict={X: X_batch, y: y_batch})

        acc_test = accuracy.eval(feed_dict={X: mnist.test.images, y: mnist.test.labels})

        print(epoch, "Train accuracy:", acc_train, "Test accuracy:", acc_test)

 

    save_path = saver.save(sess, "C:/Users/CSH/Desktop/tobigs/week9/my_model_final.ckpt")

 

0 Train accuracy: 0.89 Test accuracy: 0.8954

1 Train accuracy: 0.91 Test accuracy: 0.9199

2 Train accuracy: 0.91 Test accuracy: 0.9299

3 Train accuracy: 0.96 Test accuracy: 0.934

4 Train accuracy: 0.92 Test accuracy: 0.9414

5 Train accuracy: 0.92 Test accuracy: 0.944

6 Train accuracy: 0.96 Test accuracy: 0.9489

7 Train accuracy: 0.95 Test accuracy: 0.9522

8 Train accuracy: 0.99 Test accuracy: 0.9551

9 Train accuracy: 0.94 Test accuracy: 0.9563

10 Train accuracy: 0.98 Test accuracy: 0.9584

11 Train accuracy: 0.99 Test accuracy: 0.959

12 Train accuracy: 0.98 Test accuracy: 0.9615

13 Train accuracy: 0.98 Test accuracy: 0.9616

14 Train accuracy: 1.0 Test accuracy: 0.964

15 Train accuracy: 1.0 Test accuracy: 0.9653

16 Train accuracy: 0.99 Test accuracy: 0.9669

17 Train accuracy: 0.96 Test accuracy: 0.968

18 Train accuracy: 0.99 Test accuracy: 0.9678

19 Train accuracy: 0.98 Test accuracy: 0.9689

#이전보다 확연히 accuracy 높아졌다. 하지만 아직 목표점 98% 이르진 못했다.

 

2.  optimizer 변경

 

# 보다  좋은 성능을 위해서 optimizer Adam으로 바꿔보겠다.

 

# Cell # 18

learning_rate = 0.01

 

with tf.name_scope("train"):

    optimizer = tf.train.AdamOptimizer(learning_rate)

    training_op = optimizer.minimize(loss)

 

0 Train accuracy: 0.98 Test accuracy: 0.9605

1 Train accuracy: 0.98 Test accuracy: 0.9579

2 Train accuracy: 0.99 Test accuracy: 0.9602

3 Train accuracy: 0.97 Test accuracy: 0.9622

4 Train accuracy: 0.98 Test accuracy: 0.9672

5 Train accuracy: 1.0 Test accuracy: 0.9588

6 Train accuracy: 0.99 Test accuracy: 0.9664

7 Train accuracy: 0.95 Test accuracy: 0.9671

8 Train accuracy: 1.0 Test accuracy: 0.9695

9 Train accuracy: 0.99 Test accuracy: 0.9652

10 Train accuracy: 0.98 Test accuracy: 0.9685

11 Train accuracy: 1.0 Test accuracy: 0.9691

12 Train accuracy: 0.99 Test accuracy: 0.97

13 Train accuracy: 1.0 Test accuracy: 0.9719

14 Train accuracy: 0.99 Test accuracy: 0.9669

15 Train accuracy: 0.99 Test accuracy: 0.97

16 Train accuracy: 1.0 Test accuracy: 0.9709

17 Train accuracy: 0.99 Test accuracy: 0.9753

18 Train accuracy: 0.99 Test accuracy: 0.9724

19 Train accuracy: 1.0 Test accuracy: 0.9735

# 전체적으로 accuracy 더욱 높아졌다. 하지만 96~97%에서 높아지지가 않는다.

 

3.  L1, L2 Regularization

 

# 아까 모델을 더욱 깊게 만들었기 때문에 오버피팅 위험이 있을 있다. 이를 막기 위해 Regularization 해보자.

# 몇몇 방법이 있지만 중에서 L2 Regularization 시도해보았다.

 

# Cell # 16

with tf.name_scope("dnn"):

    hidden1 = tf.layers.dense(X, n_hidden1, name="hidden1",  kernel_initializer= he_init,

                              kernel_regularizer= tf.contrib.layers.l2_regularizer(0.01), activation=tf.nn.relu)

    hidden2 = tf.layers.dense(hidden1, n_hidden2, name="hidden2", kernel_initializer= he_init,

                              kernel_regularizer= tf.contrib.layers.l2_regularizer(0.01), activation=tf.nn.relu)

    hidden3 = tf.layers.dense(hidden2, n_hidden3, name="hidden3", kernel_initializer= he_init,

                              kernel_regularizer= tf.contrib.layers.l2_regularizer(0.01), activation=tf.nn.relu)

    hidden4 = tf.layers.dense(hidden2, n_hidden4, name="hidden4", kernel_initializer= he_init,

                              kernel_regularizer= tf.contrib.layers.l2_regularizer(0.01), activation=tf.nn.relu)

    logits = tf.layers.dense(hidden4, n_outputs, name="outputs")

 

0 Train accuracy: 0.92 Test accuracy: 0.9411

1 Train accuracy: 0.96 Test accuracy: 0.9564

2 Train accuracy: 0.98 Test accuracy: 0.96

3 Train accuracy: 0.98 Test accuracy: 0.9585

4 Train accuracy: 0.96 Test accuracy: 0.9676

5 Train accuracy: 0.98 Test accuracy: 0.9681

6 Train accuracy: 0.98 Test accuracy: 0.9663

7 Train accuracy: 0.96 Test accuracy: 0.9671

8 Train accuracy: 0.97 Test accuracy: 0.9716

9 Train accuracy: 1.0 Test accuracy: 0.9596

10 Train accuracy: 0.98 Test accuracy: 0.9673

11 Train accuracy: 0.98 Test accuracy: 0.9673

12 Train accuracy: 0.96 Test accuracy: 0.9618

13 Train accuracy: 1.0 Test accuracy: 0.9716

14 Train accuracy: 0.98 Test accuracy: 0.9693

15 Train accuracy: 0.99 Test accuracy: 0.9717

16 Train accuracy: 0.99 Test accuracy: 0.9705

17 Train accuracy: 1.0 Test accuracy: 0.9678

18 Train accuracy: 0.99 Test accuracy: 0.9692

19 Train accuracy: 0.99 Test accuracy: 0.9661

# 여전히 96~97% 내에서 왔다갔다만 뿐이었다.

# 이번엔 learning rate 커서 최적점에 도달 못하고 주변에서 멤돌다 말아버리는 것은 아닐까 추정했다

# 그래서 learning rate 낮춰보았다.

 

 

4. Learning Rate 수정

 

# 0.01에서 0.001 낮춰보겠다.

 

# Cell # 18

learning_rate = 0.001

 

with tf.name_scope("train"):

    optimizer = tf.train.AdamOptimizer(learning_rate)

    training_op = optimizer.minimize(loss)

 

0 Train accuracy: 0.97 Test accuracy: 0.963

1 Train accuracy: 0.99 Test accuracy: 0.9643

2 Train accuracy: 1.0 Test accuracy: 0.9768

3 Train accuracy: 0.98 Test accuracy: 0.9711

4 Train accuracy: 1.0 Test accuracy: 0.9749

5 Train accuracy: 1.0 Test accuracy: 0.977

6 Train accuracy: 1.0 Test accuracy: 0.9729

7 Train accuracy: 1.0 Test accuracy: 0.9817

8 Train accuracy: 1.0 Test accuracy: 0.9821

9 Train accuracy: 1.0 Test accuracy: 0.9739

10 Train accuracy: 0.99 Test accuracy: 0.9771

11 Train accuracy: 1.0 Test accuracy: 0.9778

12 Train accuracy: 1.0 Test accuracy: 0.9809

13 Train accuracy: 1.0 Test accuracy: 0.9813

14 Train accuracy: 0.99 Test accuracy: 0.9783

15 Train accuracy: 1.0 Test accuracy: 0.9782

16 Train accuracy: 1.0 Test accuracy: 0.9798

17 Train accuracy: 1.0 Test accuracy: 0.9812

18 Train accuracy: 1.0 Test accuracy: 0.9839

19 Train accuracy: 1.0 Test accuracy: 0.9807

# accuracy 98% 넘었다.


List of Articles
번호 제목 글쓴이 날짜 조회 수
공지 R 소스 공유 게시판 이용 관련 공지사항 1 DataMarket 2014.05.21 13222
92 투빅스 7&8기 9주차 과제 Neural Network를 이용한 MNIST 분류 - 8기 김민정 민정e 2017.09.23 235
» 투빅스 7&8기 9주차 과제 Neural Network를 이용한 MNIST 분류 - 8기 최서현 최서현 2017.09.22 203
90 투빅스 7&8기 7주차 과제 유기동물 과제 - 8기 조양규 dial123 2017.09.14 308
89 투빅스 7&8기 7주차 과제 유기동물입양예측 - 8기 김강열 김강열 2017.09.14 325
88 투빅스 7&8기 6주차 과제 word2vec - 8기 황다솔 다솔 2017.08.31 479
87 투빅스 7&8기 6주차 과제 TF-IDF 문서유사도 측정 - 8기 최서현 최서현 2017.08.31 367
86 투빅스 7&8기 5주차 과제 Selenium Crawling - 8기 김강열 김강열 2017.08.24 506
85 투빅스 7&8기 5주차 과제 Image Augmentation - 8기 김민정 김소희 최수정 황다솔 file 민정e 2017.08.24 458
84 투빅스 7&8기 5주차 과제 Beautiful Soup 이용한 Crawling - 8기 류호성 file 류호성 2017.08.24 456
83 투빅스 7&8기 4주차 과제 tree, RF, bagging, boosting 이용 분석 - 8기 조양규 file dial123 2017.08.17 510
82 투빅스 7&8기 4주차 과제 의사결정나무&랜덤포레스트 - 8기 김강열 김강열 2017.08.17 514
81 투빅스 7&8기 3주차 과제 클러스터링 구현 - 8기 권문정 김강열 이현경 조양규 1 이현경 2017.08.10 571
80 투빅스 7&8기 3주차 과제 PCA - 8기 이현경 file 이현경 2017.08.12 560
79 투빅스 7&8기 2주차 과제 연관성 분석 - 8기 조양규 file dial123 2017.08.03 571
78 투빅스 7&8기 2주차 과제 나이브베이즈 구현 - 8기 이현경 file 이현경 2017.08.03 531
77 투빅스 7&8기 2주차 과제 로지스틱/Ridge/Lasso&알고리즘 - 8기 김강열 file 김강열 2017.08.03 626
76 투빅스 7&8기 1주차 과제 알고리즘 - 8기 김강열 file 김강열 2017.07.27 672
75 투빅스 7&8기 1주차 과제 회귀분석 - 8기 황다솔 file 다솔 2017.07.27 820
74 투빅스 6&7기 8주차 과제 PCA(주성분 분석) - 7기 이동수 1 탱탱볼 2017.03.18 1679
73 투빅스 6&7기 8주차 과제 LBP 알고리즘 구현 - 7기 이광록 1 file 2017.03.16 1423
Board Pagination ‹ Prev 1 2 3 4 ... 5 Next ›
/ 5

나눔글꼴 설치 안내


이 PC에는 나눔글꼴이 설치되어 있지 않습니다.

이 사이트를 나눔글꼴로 보기 위해서는
나눔글꼴을 설치해야 합니다.

설치 취소

Designed by sketchbooks.co.kr / sketchbook5 board skin

Sketchbook5, 스케치북5

Sketchbook5, 스케치북5

Sketchbook5, 스케치북5

Sketchbook5, 스케치북5