apaam/mlpack
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Squashed commit of the following:

Browse Source 
commit 54c6ebe03a
Merge: 775a3b55f b406fc150
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun May 1 13:24:24 2022 -0400

    Merge pull request #3200 from shubham1206agra/go-cli-fix

    Go Build Fix

commit 775a3b55f7
Merge: 8e72ed698 54e291443
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 30 10:41:45 2022 -0400

    Merge pull request #3198 from shubham1206agra/py-cli-fix

    Python Build Fix

commit 8e72ed6986
Merge: f7cd03866 1c1182301
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 30 10:40:59 2022 -0400

    Merge pull request #3199 from shubham1206agra/r-cli-fix

    R Build Fix

commit b406fc1506
Author: shubham1206agra <tt1191044@iitd.ac.in>
Date:   Fri Apr 29 10:42:32 2022 +0530

    changes according to suggestion

commit 54e2914430
Author: Shubham Agrawal <58412969+shubham1206agra@users.noreply.github.com>
Date:   Fri Apr 29 10:20:14 2022 +0530

    Update src/mlpack/bindings/python/copy_artifacts.py

    Co-authored-by: Ryan Curtin <ryan@ratml.org>

commit f7cd038660
Merge: 065fcee29 3eb8ae67e
Author: Ryan Curtin <ryan@ratml.org>
Date:   Thu Apr 28 19:00:58 2022 -0700

    Merge pull request #2777 from zoq/ann-vtable

    Swap boost::variant with vtable.

commit ad4569213a
Author: shubham1206agra <tt1191044@iitd.ac.in>
Date:   Thu Apr 28 19:32:14 2022 +0530

    temp sol to version issue

commit 1bfa385663
Author: shubham1206agra <tt1191044@iitd.ac.in>
Date:   Thu Apr 28 18:48:03 2022 +0530

    initial fix by disabling go modules

commit 1c1182301e
Author: shubham1206agra <tt1191044@iitd.ac.in>
Date:   Thu Apr 28 12:46:18 2022 +0530

    force install pkgbuild

commit 3cfbb4e65d
Author: shubham1206agra <tt1191044@iitd.ac.in>
Date:   Thu Apr 28 12:36:51 2022 +0530

    cleanup

commit feab906927
Author: shubham1206agra <tt1191044@iitd.ac.in>
Date:   Thu Apr 28 11:28:32 2022 +0530

    missing '/' added

commit 577cc2d692
Author: shubham1206agra <tt1191044@iitd.ac.in>
Date:   Thu Apr 28 10:33:02 2022 +0530

    new directory structure using glob

commit 3eb8ae67ec
Merge: 6f98ab7bc 065fcee29
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Apr 27 20:51:46 2022 -0400

    Merge remote-tracking branch 'origin/master' into ann-vtable

commit 6f98ab7bc9
Author: Eshaan Agarwal <eshaan060202@gmail.com>
Date:   Wed Apr 13 20:13:06 2022 +0530

    Fix style issues

    Co-authored-by: Ryan Curtin <ryan@ratml.org>

commit ac3097e875
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Wed Apr 13 18:22:03 2022 +0530

    fix error in size_t cast

    Signed-off-by: eshaanagarwal <eshaan060202@gmail.com>

commit d03aafacf6
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Tue Apr 5 13:02:00 2022 +0530

    add parameter documentation in size checks

commit e55609d5be
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Tue Apr 5 00:51:27 2022 +0530

    Add transpose parameter in size check

commit a485da2b71
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Thu Mar 31 04:22:53 2022 +0530

    fixed issues in styling

commit dff01492ed
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Mon Mar 14 22:25:27 2022 +0530

    fixed styling issues

commit ca50361083
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Fri Mar 11 22:50:03 2022 +0530

    fix build issue by removing row vector assert condition

commit 5b7d36db85
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Fri Mar 11 20:15:07 2022 +0530

    fix failed build

commit 614f924a4e
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Fri Mar 11 19:50:54 2022 +0530

    fixed redundancy in size-checks

commit f67c566a5f
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Fri Mar 11 19:30:22 2022 +0530

    fix matrix completion size-checks

commit e1e2b5308b
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Fri Mar 11 12:19:37 2022 +0530

    remove incorrect checks in adaboost

commit e5d701df32
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Thu Mar 10 16:30:49 2022 +0530

    fix size checks

commit 2656b300f9
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Thu Mar 10 12:24:15 2022 +0530

    fix styling issue

    Signed-off-by: eshaanagarwal <eshaan060202@gmail.com>

commit ef4d293fed
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Wed Mar 9 02:33:08 2022 +0530

    Add: size checks for kmeans and linear regression

commit 640dd0cde8
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Tue Mar 8 02:27:38 2022 +0530

    Add : Size checks for adaboost and matix completion

    Signed-off-by: eshaanagarwal <eshaan060202@gmail.com>

commit 66bc9cbe00
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 15 21:29:30 2022 -0400

    Huh, I guess it is a new year.

commit 7b0c1e157b
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 15 21:29:15 2022 -0400

    Update HISTORY.

commit e4be17defe
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 15 21:26:42 2022 -0400

    Add test for KFoldCV and Perceptron.

commit a70437ffc3
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 15 21:26:28 2022 -0400

    Add constructor to Perceptron for weighted data for KFoldCV.

commit 9b86271002
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 15 21:26:08 2022 -0400

    Make Classify() set the output predictions' size.

commit a35fc3b994
Author: Yashwants19 <Yashwants19@users.noreply.github.com>
Date:   Sun Apr 17 10:02:24 2022 +0000

    Upgrade Catch to 2.13.9

commit a1bb763729
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Apr 16 21:40:15 2022 +0200

    Update src/mlpack/core/data/save_image.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit 57cec6f081
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Apr 16 16:20:14 2022 +0100

    Apply @rcurtin modification to check STB version

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 8e34862cf6
Author: Omar Shrit <omar@shrit.me>
Date:   Sun Mar 13 21:28:32 2022 +0000

    Let us see inital try for STB test

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit afcc862ced
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Feb 3 21:46:40 2022 +0000

    Adding the missing starts, shitty regexp

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 0c076cca19
Author: Omar Shrit <omar@shrit.me>
Date:   Tue Jan 25 12:09:08 2022 +0000

    Finish this PR

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit b26f2d2a15
Author: Omar Shrit <omar@shrit.me>
Date:   Mon Jan 24 22:15:55 2022 +0000

    Refactor save_image into save_image_impl finally done!!

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 8fffbd55fa
Author: Omar Shrit <omar@shrit.me>
Date:   Mon Jan 24 21:20:12 2022 +0000

    Adjust namespace of mlpack::Log

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 37367c7ec5
Author: Omar Shrit <omar@shrit.me>
Date:   Mon Jan 24 19:51:55 2022 +0100

    Update src/mlpack/core/data/image_info_impl.hpp

    Co-authored-by: Ryan Curtin <ryan@ratml.org>

commit 3cc2d250b5
Author: Omar Shrit <omar@shrit.me>
Date:   Mon Jan 24 19:51:49 2022 +0100

    Update src/mlpack/core/data/image_info_impl.hpp

    Co-authored-by: Ryan Curtin <ryan@ratml.org>

commit 55c6cfd7df
Author: Omar Shrit <omar@shrit.me>
Date:   Sun Jan 23 19:38:29 2022 +0000

    Apply @rcurtin patch to fix the STB issue.

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 8cdb53a313
Author: Omar Shrit <omar@shrit.me>
Date:   Sun Jan 16 18:20:04 2022 +0000

    Commenting all #defines that are causing the problems

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit e2e000b06f
Author: Omar Shrit <omar@shrit.me>
Date:   Sun Jan 16 16:21:27 2022 +0000

    Move constructor to implementation

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 981f57c4be
Author: Omar Shrit <omar@shrit.me>
Date:   Sun Jan 16 16:03:30 2022 +0000

    Compiling locally, adding all modifications

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 4d29f5f9c1
Author: Omar Shrit <omar@shrit.me>
Date:   Sun Jan 16 12:59:31 2022 +0000

    Provide more details for namespace in src/mlpack/methods/det/dtree_impl.hpp

    Co-authored-by: Ryan Curtin <ryan@ratml.org>

commit e8a76f8f35
Author: Omar Shrit <omar@shrit.me>
Date:   Sun Jan 16 12:59:03 2022 +0000

    Remove mlpack namspace from src/mlpack/methods/hmm/hmm_util_impl.hpp

    Co-authored-by: Ryan Curtin <ryan@ratml.org>

commit 9dd15d0865
Author: Omar Shrit <omar@shrit.me>
Date:   Sun Jan 16 12:58:50 2022 +0000

    remove mlpack namespace from src/mlpack/methods/reinforcement_learning/q_networks/categorical_dqn.hpp

    Co-authored-by: Ryan Curtin <ryan@ratml.org>

commit aa057ccb8b
Author: Omar Shrit <omar@shrit.me>
Date:   Sun Jan 16 12:58:38 2022 +0000

    Remove mlpack namespace from src/mlpack/methods/hmm/hmm_util_impl.hpp

    Co-authored-by: Ryan Curtin <ryan@ratml.org>

commit 61cc69b1a5
Author: Omar Shrit <omar@shrit.me>
Date:   Sun Jan 16 12:58:22 2022 +0000

    Remove mlpack namespace from src/mlpack/methods/reinforcement_learning/q_networks/dueling_dqn.hpp

    Co-authored-by: Ryan Curtin <ryan@ratml.org>

commit 31f9545685
Author: Omar Shrit <omar@shrit.me>
Date:   Sun Jan 16 12:58:06 2022 +0000

    Remove mlpack namespace from src/mlpack/methods/reinforcement_learning/q_networks/simple_dqn.hpp

    Co-authored-by: Ryan Curtin <ryan@ratml.org>

commit 665711b9ce
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 16 15:33:51 2021 +0100

    Remove additional line in src/mlpack/core/kernels/pspectrum_string_kernel.hpp

    Co-authored-by: Ryan Curtin <ryan@ratml.org>

commit 895c885b44
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 16 15:33:08 2021 +0100

    Fix indentation in src/mlpack/core/data/detect_file_type.hpp

    Co-authored-by: Ryan Curtin <ryan@ratml.org>

commit cf6f60aee8
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 16 15:32:55 2021 +0100

    Add forgetten dot in src/mlpack/core/math/random_basis.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit 21e045928a
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 16 15:32:38 2021 +0100

    Add spaces in src/mlpack/core/math/lin_alg_impl.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit 1f79e4fc83
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 16 15:32:16 2021 +0100

    Remove additional line in src/mlpack/core/math/lin_alg_impl.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit 6dc2d63af6
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 16 15:32:00 2021 +0100

    Fix indentation src/mlpack/core/math/lin_alg_impl.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit ed07d9f05a
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 16 15:31:41 2021 +0100

    Fix indentation in src/mlpack/core/math/lin_alg_impl.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit 047dc47b3b
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 16 15:31:24 2021 +0100

    Add parantheses in src/mlpack/core/math/lin_alg_impl.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit 2dae84947a
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 16 15:31:00 2021 +0100

    Fix style in src/mlpack/core/kernels/epanechnikov_kernel_impl.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit 8666165293
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 16 15:30:43 2021 +0100

    Update style in src/mlpack/core/kernels/epanechnikov_kernel_impl.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit 6254b4b49a
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 2 23:54:27 2021 +0000

    Add forgetten layer name

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 407b563330
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 2 22:54:06 2021 +0000

    Clean all using namespaces from the headers.

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 4a713818ab
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 2 19:40:02 2021 +0000

    Fix the random_basis_impl in addition

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 99a485b6be
Author: Omar Shrit <omar@shrit.me>
Date:   Thu Dec 2 19:39:05 2021 +0000

    Clean partly the namespace. Let us see if this resolve the macOS issues

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 986bdb6bdd
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 20 22:03:31 2021 +0000

    Adjust namespace in preprocess

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit c3691bdf65
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 20 20:33:55 2021 +0000

    Remove the mlpack:: namespace

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit adbedb857b
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 20 19:56:14 2021 +0000

    Adding missing math/random headers

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit d246f94286
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 20 19:29:07 2021 +0000

    Fix the gmm error

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 10ed5c1c96
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 20 17:06:11 2021 +0000

    Let us if this resolves the binding issues

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 8b0c8038dd
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 13 19:52:19 2021 +0000

    Adding a missing std string

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit ff2f37c4e7
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 13 19:40:23 2021 +0000

    Finish the pspectrum_string_kernel

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit c6bf80178a
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 13 19:39:32 2021 +0000

    Deleting epanechnikov_kernel and adding pspectrum_string_kernel

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 09f508c6de
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 13 19:33:05 2021 +0000

    Finishing the epanechnikov_kernel

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 61d01a3949
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 13 19:23:57 2021 +0000

    remove epanechnikov_kernel

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit bd84bb2d0d
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 13 19:15:58 2021 +0000

    inline random_basis

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 90b80ea766
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 13 19:11:32 2021 +0000

    Adding missing headers

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit e0c5ac34fb
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 13 18:56:00 2021 +0000

    move the impl from .cpp to .impl

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 85a53b486f
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 13 18:43:17 2021 +0000

    Fix the compilation warning related to include implementations

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit c1d3833ff1
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 13 18:24:15 2021 +0000

    Fix comments and compilation bugs

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit b037f4f69f
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 13 18:12:02 2021 +0000

    Finish inlining the data dir

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit 0941f65038
Author: Omar Shrit <omar@shrit.me>
Date:   Sat Nov 13 18:04:01 2021 +0000

    Make save image header only

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit ba84188163
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Wed Mar 16 23:23:01 2022 -0400

    Prettify C++ code used to check for atomic linkage.

commit f9cc2f6bfa
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Tue Mar 15 22:14:12 2022 -0400

    If we use MSVC no need to check for atomic.

commit 119da780ed
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Feb 20 21:15:50 2022 -0500

    Check if libatomic is bundled.

commit 96f4d9aea6
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Wed Feb 16 21:42:44 2022 -0500

    Check if atomics need -latomic linking.

commit 47f94fbaee
Author: zoq <zoq@users.noreply.github.com>
Date:   Thu Apr 14 10:05:15 2022 +0000

    Upgrade Boost Version in CMake script.

commit c31961bc46
Author: LiuZhuojin <zhuojinliu.cs@gmail.com>
Date:   Thu Apr 14 09:53:55 2022 +0800

    Update contributor list

commit 2d77be64d9
Author: LiuZhuojin <zhuojinliu.cs@gmail.com>
Date:   Wed Apr 13 09:33:03 2022 +0800

    Replace boost::heap::priority_queue with std::vector

commit aa4a8b3922
Author: LiuZhuojin <zhuojinliu.cs@gmail.com>
Date:   Wed Apr 13 00:40:08 2022 +0800

    Add vector and queue to standard includes

commit d9145891a7
Author: LiuZhuojin <zhuojinliu.cs@gmail.com>
Date:   Wed Apr 13 00:38:40 2022 +0800

    Replace boost::heap::priority_queue with std::vector

commit 58518ab811
Author: LiuZhuojin <zhuojinliu.cs@gmail.com>
Date:   Wed Apr 13 00:27:28 2022 +0800

    Replace boost::heap::priority_queue with std::vector

commit bc30907f5a
Author: LiuZhuojin <zhuojinliu.cs@gmail.com>
Date:   Sat Apr 9 15:04:31 2022 +0800

    Replace boost::heap::priority_queue with std::vector

commit e117627c7f
Author: LiuZhuojin <zhuojinliu.cs@gmail.com>
Date:   Sat Apr 9 14:58:28 2022 +0800

    Replace boost::heap::priority_queue with std::vector

commit 4d962ebab6
Author: LiuZhuojin <zhuojinliu.cs@gmail.com>
Date:   Fri Apr 8 13:53:10 2022 +0800

    Replace boost::heap::priority_queue with std::vector

commit 65afadc4af
Author: Yashwants19 <Yashwants19@users.noreply.github.com>
Date:   Fri Apr 1 10:00:54 2022 +0000

    Upgrade CLI11 to 2.2.0

commit 5875d3625a
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Apr 27 16:10:15 2022 -0400

    Apply suggestions from code review

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit fef34ea8db
Author: shubham1206agra <tt1191044@iitd.ac.in>
Date:   Wed Apr 27 19:25:31 2022 +0530

    trying something else

commit 64e1f75b65
Author: shubham1206agra <tt1191044@iitd.ac.in>
Date:   Wed Apr 27 19:11:02 2022 +0530

    trying config file

commit 065fcee296
Merge: 56d3f1636 01fa1241c
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Apr 18 15:43:03 2022 -0700

    Merge pull request #3164 from eshaanagarwal/size-checks

    Added Size checks for Matrix Completion, Kmeans and Linear Regression

commit 56d3f16368
Merge: 2994570a0 152094dbb
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Apr 18 06:01:10 2022 -0700

    Merge pull request #3190 from mlpack/fix-perceptron-cv

    Add weighted data constructor to `Perceptron`

commit 2994570a05
Merge: fdc7af7a6 c47cebdd8
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Apr 18 05:55:12 2022 -0700

    Merge pull request #3191 from mlpack/catch-header-updates-2.13.9

    Upgrade Catch to 2.13.9

commit c47cebdd82
Author: Yashwants19 <Yashwants19@users.noreply.github.com>
Date:   Sun Apr 17 10:02:24 2022 +0000

    Upgrade Catch to 2.13.9

commit 1645bb22eb
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 16 21:05:15 2022 -0400

    Use `typename` instead of `class` for consistency.

commit d8a1c27b7f
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 16 20:55:50 2022 -0400

    Add serialization to GlorotInit.

commit 152094dbbe
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 15 21:29:30 2022 -0400

    Huh, I guess it is a new year.

commit c17e9c2f0f
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 15 21:29:15 2022 -0400

    Update HISTORY.

commit 9a6551a1b5
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 15 21:26:42 2022 -0400

    Add test for KFoldCV and Perceptron.

commit 3133a4abbd
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 15 21:26:28 2022 -0400

    Add constructor to Perceptron for weighted data for KFoldCV.

commit 80e65d8a5d
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 15 21:26:08 2022 -0400

    Make Classify() set the output predictions' size.

commit 01fa1241ce
Author: Eshaan Agarwal <eshaan060202@gmail.com>
Date:   Wed Apr 13 20:13:06 2022 +0530

    Fix style issues

    Co-authored-by: Ryan Curtin <ryan@ratml.org>

commit cb35eba204
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Wed Apr 13 18:22:03 2022 +0530

    fix error in size_t cast

    Signed-off-by: eshaanagarwal <eshaan060202@gmail.com>

commit f12d714c50
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Apr 11 13:34:43 2022 -0400

    Comment on the weightsPtr parameter.

commit 44b30019f4
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Apr 11 13:32:39 2022 -0400

    Add another padding test for Convolution.

commit 1a9f7ccd5b
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Apr 11 11:56:39 2022 -0400

    Return correct type of Weights().

commit bacd18db55
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Apr 11 11:53:54 2022 -0400

    Update src/mlpack/methods/ann/layer/concatenate_impl.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit 3dfe5a000d
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Apr 11 11:53:40 2022 -0400

    Update src/mlpack/methods/ann/layer/concatenate.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit 6b47a9cd67
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Apr 11 11:53:33 2022 -0400

    Update src/mlpack/methods/ann/layer/alpha_dropout.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit 77203b84cf
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Apr 11 11:53:25 2022 -0400

    Update src/mlpack/methods/ann/layer/alpha_dropout.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit 845a5a0aed
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Tue Apr 5 13:02:00 2022 +0530

    add parameter documentation in size checks

commit 07be7313a7
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Apr 4 22:23:36 2022 -0400

    Use network.Forward() to avoid the extra output copy.

commit 0710d9f111
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Apr 4 22:05:35 2022 -0400

    Update src/mlpack/methods/ann/ffn.hpp

    Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

commit 0e9f05adbc
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Tue Apr 5 00:51:27 2022 +0530

    Add transpose parameter in size check

commit 2cd38b3c9e
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 22:02:45 2022 -0400

    Some additional cleanups.

commit c38625f8d4
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 22:01:13 2022 -0400

    Change 'rho' to 'bpttSteps' for clarity.

commit 6eecba768a
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 21:55:50 2022 -0400

    Include numeric header for std::accumulate.

commit 12287174e4
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 21:29:51 2022 -0400

    Fix TODOs in LSTM layer.

commit 71ecddf8f6
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 20:49:03 2022 -0400

    Some cleanups of the RNN class.

commit 83a178dffa
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 16:53:25 2022 -0400

    Some additional type fixes.

commit f5b3885998
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 16:30:02 2022 -0400

    Remove unused typedef.

commit eae99cfaca
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 16:27:29 2022 -0400

    Simplify Shuffle() implementation.

commit 250ab5cdd0
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 16:24:42 2022 -0400

    Remove unused Swap() function declaration.

commit 1432420917
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 16:23:28 2022 -0400

    Use MakeAlias() in the RNN implementation.

commit 03045b0235
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 16:08:59 2022 -0400

    Fix line wrap.

commit fe8a288613
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 16:07:01 2022 -0400

    Minor fixes to tests.

commit 44a18de28c
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 15:58:36 2022 -0400

    Clarify RNN comment.

commit 10d04c4153
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 15:57:02 2022 -0400

    Add clarifying comment about where MakeAlias() is used.

commit 19c60244a7
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 15:56:06 2022 -0400

    More CEREAL_NVP() fixes.

commit d45d8787b9
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 15:54:58 2022 -0400

    Fix serialization: use CEREAL_NVP().

commit 194211feec
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 15:53:45 2022 -0400

    Fixes for RBFType implementation comments.

commit 9c33d6b780
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 15:52:02 2022 -0400

    Fix header guard name.

commit 880c95bee5
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 15:50:50 2022 -0400

    Fix typo.

commit 2fdd4b823b
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 15:50:27 2022 -0400

    Add explanatory comment.

commit 416dfc7c4a
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 15:48:36 2022 -0400

    Fix inaccurate comment.

commit 19563e3f63
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 14:49:37 2022 -0400

    Update comment.

commit ff3717bc53
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 14:41:01 2022 -0400

    Fix typo in comment.

commit ac42ea450b
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 14:40:39 2022 -0400

    Clarify implementations.

commit 20c7b5a836
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 14:37:01 2022 -0400

    Remove unused method.

commit 4fb0dd4965
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 14:36:36 2022 -0400

    Fix includes.

commit 5253b1427a
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 14:28:42 2022 -0400

    Clarify comment.

commit f0aceda809
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 14:27:38 2022 -0400

    Fix serialization for base layer.

commit 2413a1cdf5
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 14:26:38 2022 -0400

    Add a comment.

commit 2f0a9419ef
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 14:26:05 2022 -0400

    Use MakeAlias() instead.

commit c599ecfd1a
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 13:23:32 2022 -0400

    Change BaseLayer names for consistency.

commit 6d69ea45f1
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 13:18:45 2022 -0400

    Clear member that is not serialized on loading.

commit 8f52887fb5
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 13:16:29 2022 -0400

    Remove inaccurate comments (we did not drop these).

commit cb952537e2
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 13:15:56 2022 -0400

    Re-add removed files from CMake.

commit c75d03ff6f
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 13:14:53 2022 -0400

    Fix line spacing.

commit 8b2e57d32e
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 13:13:46 2022 -0400

    Add a clarifying comment.

commit 454c746491
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 13:12:52 2022 -0400

    Remove duplicated `training` member.

commit c6443a43d8
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 13:05:37 2022 -0400

    Add clarifying comment.

commit 6258a8a9fc
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 13:02:25 2022 -0400

    Clean up copy and move operators.

commit 07fc52fba3
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 12:51:28 2022 -0400

    Some clarifying comments.

commit 0ed1971ce0
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 12:50:24 2022 -0400

    Some cleanups of Forward() and Backward().

commit cade46d605
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 12:31:27 2022 -0400

    Some extra paranoia about `Network()`.

commit 20bf26651c
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Apr 3 12:18:34 2022 -0400

    Use a separate file for forward declarations.

commit 56ef5ded8a
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 21:38:09 2022 -0400

    Oops, revert unintentional CMake changes.

commit a1e3a8dbef
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 21:28:20 2022 -0400

    Fix minor merge issues.

commit f52a977dac
Merge: 835499fdb c4bb721ef
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 19:15:59 2022 -0400

    Merge remote-tracking branch 'origin/master' into ann-vtable

commit 835499fdb4
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 18:39:16 2022 -0400

    Fix comments and remove working comments.

commit 2950e1ff9d
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 18:36:29 2022 -0400

    Add comments to RecurrentLayer implementation.

commit 23d27c9b5d
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 18:20:59 2022 -0400

    Minor style fixes.

commit e542fd8404
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 18:19:44 2022 -0400

    Comment default typedefs.

commit 59ca9428c6
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 18:16:28 2022 -0400

    Add standardized comment about MatType.

commit cc4e3aec28
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 17:52:48 2022 -0400

    Use 'Type' and typedef conventions for loss functions.

commit 232ce2a342
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 15:37:55 2022 -0400

    Adapt CustomLayer for testing to have only one template parameter.

commit 9a9b6334a6
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 15:37:41 2022 -0400

    Adapt tests for FFNs and RNNs having only one template parameter for type.

commit 9fe42a2d7b
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 15:37:21 2022 -0400

    Adapt to use only one template parameter.

commit 6ff20e912b
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 15:36:52 2022 -0400

    Adapt to use only one template parameter for types.

commit 48685d3483
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Apr 2 15:36:34 2022 -0400

    Adapt to use only one template parameter.

commit d79df40382
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 1 19:42:59 2022 -0400

    Don't allow calling Parameters() on a layer with no weights.

commit 6cdd49ab88
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 1 18:21:50 2022 -0400

    Some early attempts to adapt some tests.

commit 0d73c5e1e2
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 1 18:20:50 2022 -0400

    Early attempts at refactoring Q-learning code.

commit a446c67f5a
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Apr 1 17:53:47 2022 -0400

    Re-enable RLComponentsTest.

commit 263d1ddd15
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Mar 30 22:07:37 2022 -0400

    Allow the Rho parameter to be reset (it should also be renamed).

commit bcbeebcf27
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Mar 30 22:07:27 2022 -0400

    Adapt CustomLayer.

commit 53d903d1c4
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Mar 30 21:17:29 2022 -0400

    Re-add the callback tests.

commit f34513341f
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Mar 30 21:15:36 2022 -0400

    We don't have an arbitrary type; move() should be used here.

commit 5a2f58f559
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Mar 30 20:03:55 2022 -0400

    Re-enable some of the activation function tests.

commit 0155643e72
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Mar 30 19:36:28 2022 -0400

    Forward() should not worry about whether we are in single mode (that's only relevant for training).

commit b3b5c6827a
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Thu Mar 31 04:22:53 2022 +0530

    fixed issues in styling

commit ed424ae4a1
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Mar 29 21:23:37 2022 -0400

    Remove other bits of boost::visitor things.

commit be9a3882ad
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Mar 29 21:23:28 2022 -0400

    Comment out RBMNetworkTests.

commit f1f5fc7fa2
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Mar 29 21:23:22 2022 -0400

    Fix includes for moved files.

commit 112016ccf1
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Mar 29 21:22:59 2022 -0400

    Remove boost usage.

commit 2da71b1f36
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Mar 29 21:22:46 2022 -0400

    Move some files that aren't yet adapted or are unneeded.

commit 1998227d9c
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Mar 29 19:58:28 2022 -0400

    Fix header guard name.

commit f2a1b79e99
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Mar 27 14:16:58 2022 -0400

    Fix incorrect step reference.

commit 7fa7eb15f0
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Mar 27 12:16:33 2022 -0400

    Don't forget virtual destructor for LSTMType.

commit a501389238
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Mar 27 12:16:14 2022 -0400

    Make sure to be able to serialize RecurrentLayer.

commit 4c8364c169
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Mar 27 12:15:52 2022 -0400

    No need to serialize the training-time-only predictors and responses.

commit c219a57da3
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Mar 27 12:15:38 2022 -0400

    Reference local `results` instead of class-wide `responses`.

commit 9b7b7f875d
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Mar 27 12:15:20 2022 -0400

    Add copy and move operator implementation.

commit accce42b35
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Mar 27 12:14:55 2022 -0400

    Oops, add RecurrentLayer to the repository.

commit df613499f3
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Mar 25 11:59:49 2022 -0400

    Adapt test to use multiple epochs.

commit aa0ce85656
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Mar 25 11:58:43 2022 -0400

    Fix alias computation.

commit 20b109fe42
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Mar 25 11:57:57 2022 -0400

    Set step correctly for forward pass.

commit 9ff1fb11f5
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Mar 23 22:40:52 2022 -0400

    Test that the RNN and FFN give the same output for only one time step.

commit 66d929f923
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Mar 23 22:40:42 2022 -0400

    Handle series with only one time step.

commit 0c97b42d80
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Mar 23 17:58:28 2022 -0400

    Be sure to serialize rho and single too.

commit 76992e86d6
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Mar 22 21:26:19 2022 -0400

    Add serialization for some more initializations.

commit d6e200ff37
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Mar 22 21:26:09 2022 -0400

    Fix include issue (this may not be the right fix).

commit 080dcfccb4
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Mar 22 21:25:59 2022 -0400

    Correct NumFunctions() implementation.

commit 4816d7ccfe
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Mar 19 23:48:10 2022 -0400

    Some other minor bugfixes; now the tests pass.

commit 80efe31e5b
Author: Ryan Curtin <ryan@ratml.org>
Date:   Thu Mar 17 22:06:01 2022 -0400

    Fix a few bugs---now LSTMBatchSizeTest works!
    (That doesn't mean that RNNs actually work...)

commit eab019541d
Author: Ryan Curtin <ryan@ratml.org>
Date:   Thu Mar 17 21:12:19 2022 -0400

    Comment out tests that don't work yet.

commit 9ff8fd401d
Author: Ryan Curtin <ryan@ratml.org>
Date:   Thu Mar 17 21:11:47 2022 -0400

    Clean up (in some places) RNN implementation.

commit 5683066a66
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Mon Mar 14 22:25:27 2022 +0530

    fixed styling issues

commit 68e40a6b23
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Fri Mar 11 22:50:03 2022 +0530

    fix build issue by removing row vector assert condition

commit ca353778a4
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Fri Mar 11 20:15:07 2022 +0530

    fix failed build

commit 6883c30cb2
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Fri Mar 11 19:50:54 2022 +0530

    fixed redundancy in size-checks

commit 2685d69097
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Fri Mar 11 19:30:22 2022 +0530

    fix matrix completion size-checks

commit f4db66192b
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Fri Mar 11 12:19:37 2022 +0530

    remove incorrect checks in adaboost

commit 05d74baffd
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Thu Mar 10 16:30:49 2022 +0530

    fix size checks

commit 54f6506015
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Thu Mar 10 12:24:15 2022 +0530

    fix styling issue

    Signed-off-by: eshaanagarwal <eshaan060202@gmail.com>

commit 8714c77a95
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Wed Mar 9 02:33:08 2022 +0530

    Add: size checks for kmeans and linear regression

commit 0a0681f2a5
Author: eshaanagarwal <eshaan060202@gmail.com>
Date:   Tue Mar 8 02:27:38 2022 +0530

    Add : Size checks for adaboost and matix completion

    Signed-off-by: eshaanagarwal <eshaan060202@gmail.com>

commit bd08f9a92e
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Feb 16 22:18:54 2022 -0500

    Remove debugging output.

commit d8b72a862b
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Wed Feb 9 22:14:32 2022 -0500

    Some RNN refactoring.

commit 93fecce9c8
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Feb 15 23:00:08 2022 -0500

    Significant cleanup of all layers.

    The ones not in not_adapted/ (other than the LSTM) are ready for review.

commit ce12cd3e25
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Feb 9 22:44:07 2022 -0500

    Remove files from CMakeLists.txt.

commit ee117f355b
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Feb 9 22:43:38 2022 -0500

    Remove layer_traits.hpp since it's no longer needed.

commit 7b969f3014
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Feb 9 22:41:54 2022 -0500

    Move two layers that got missed.

commit e2502cbb9c
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Feb 9 22:41:09 2022 -0500

    Move unadapted layers into a separate directory (for organization).

commit f72340106c
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Feb 9 22:09:08 2022 -0500

    Add some documentation about what will happen.

commit 637b93ddc0
Merge: 2065e17df cf190e11f
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Feb 9 22:05:56 2022 -0500

    Merge remote-tracking branch 'origin/master' into ann-vtable

commit 2065e17df8
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Feb 9 21:45:29 2022 -0500

    Add a test for uneven stride.

commit c532632ce9
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Feb 5 20:41:43 2022 -0500

    Remove comments that turn out to be unnecessary to address.

commit 5e09ed1671
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Feb 5 18:08:06 2022 -0500

    Fix implementations of NaiveConvolution for stride and dilation.

commit 5cd0e38eb6
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Feb 5 18:07:39 2022 -0500

    Add tests for different strides and dilations.

commit 8c1a818540
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Feb 2 22:55:35 2022 -0500

    Fix stride usage for backwards pass.

commit 110ed75b6f
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Feb 1 18:06:55 2022 -0500

    Fix some minor issues and merge problems.

commit a77d290214
Author: Ryan Curtin <ryan@ratml.org>
Date:   Thu Jan 27 16:28:22 2022 -0500

    Add/fix final set of copy/move constructors/operators.

commit 11cc9a122b
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jan 26 22:29:45 2022 -0500

    Add and fix a bunch more copy/move constructors/operators.

commit ec06bcbbff
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Jan 25 22:44:34 2022 -0500

    Start implementing copy and move constructors correctly.

commit f842111039
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Jan 25 22:44:25 2022 -0500

    Set sizes correctly for test.

commit 3f5e06e9c5
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Jan 22 11:20:35 2022 -0500

    Restructure PaddingTest for slight behavior changes.

commit abb1450102
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Jan 21 23:32:35 2022 -0500

    Fix minor bugs in shape computation.

commit d600ece415
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Jan 21 23:03:24 2022 -0500

    Fix failing padding tests.

commit 94b3294a5d
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Jan 21 14:40:53 2022 -0500

    Fix some merge issues.

commit c3caed0490
Merge: 09a28caa9 3264cd87e
Author: Ryan Curtin <ryan@ratml.org>
Date:   Thu Jan 20 20:23:26 2022 -0500

    Merge remote-tracking branch 'origin/master' into ann-vtable

commit 09a28caa90
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jan 19 17:41:18 2022 -0500

    Some minor bugfixes to FFN.  Some need further cleanup.

commit 96f97421f2
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jan 19 17:16:57 2022 -0500

    Fix bug in Gradient() implementation.

commit 6dc0596ad6
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jan 19 17:16:44 2022 -0500

    Consider bias term in gradient.

commit 9a71a61aac
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Jan 7 18:16:18 2022 -0500

    Fix setting of inputDimensionsAreSet.

commit a4ee2545f6
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Jan 4 20:12:34 2022 -0500

    Add a sanity check test.  It passes.

commit f5c4b95860
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Nov 28 21:47:40 2021 -0500

    Remove debugging output.

commit d8fef7eed6
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Nov 28 21:47:17 2021 -0500

    Adapt to new mlpack 4 conventions.

commit bda6e0969f
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Nov 28 21:46:59 2021 -0500

    Fix failing tests.

commit 8d1c05c36f
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Nov 28 21:46:44 2021 -0500

    Update to mlpack 4 conventions.

commit ead043c9b8
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Nov 28 21:45:41 2021 -0500

    Oops, make sure to add the bias in the forward pass.

commit eae19f7c28
Author: Omar Shrit <omar@shrit.me>
Date:   Tue Nov 16 17:43:09 2021 +0000

    Finishing the Alphadroput layer, tests are passing

    Signed-off-by: Omar Shrit <omar@shrit.me>

commit c19e39a152
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Nov 14 22:30:39 2021 -0500

    Turns out the MaxPooling adaptation I did was wrong---this seems more correct.

commit a1fd4829a1
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Nov 14 22:30:15 2021 -0500

    Redo Convolution layer implementation.  I think this is right but not 100% sure.

commit ca010221e9
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Nov 14 22:29:26 2021 -0500

    Enable LeakyReLU layer.

commit 49954574e9
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Nov 1 19:17:24 2021 -0400

    Add gradient test for Convolution layer... but it seems to work?

commit be5b4e5e01
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Oct 20 17:46:03 2021 -0400

    Fix shape of input to reflect the number of input maps.

commit 19f06709d4
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Oct 19 10:18:14 2021 -0400

    Use MakeAlias() to avoid accidental copies.

commit 081c32593d
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Sep 28 11:35:41 2021 -0400

    Bias should be one per output map.

commit 58f1718be1
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Sep 28 11:22:07 2021 -0400

    Huh, it seems like this fixes ConvolutionLayerPaddingTest.

commit 6fff4457c6
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Sep 20 11:09:36 2021 -0400

    Fix two more tests by making sure the inputs and outputs are right.

commit ad2cb74d05
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Sep 19 00:10:20 2021 -0400

    Fix some tests.

commit 051205d78f
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Sep 15 22:12:20 2021 -0400

    Now at least the tests don't segfault. :)

commit 1152c75d23
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Sep 15 13:59:43 2021 -0400

    Update some ann_layer tests.

commit f05590b878
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Sep 15 13:59:29 2021 -0400

    Fix convolution.

commit 1cbdaa8ac5
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Sep 15 13:58:09 2021 -0400

    Fix bug for uninitialized output.

commit 3a00d82a23
Author: Ryan Curtin <ryan@ratml.org>
Date:   Thu Sep 9 12:02:31 2021 -0400

    Some cleanups for the convolution layer.

commit 72c0f58a27
Author: Ryan Curtin <ryan@ratml.org>
Date:   Thu Sep 9 12:02:06 2021 -0400

    Fix some incorrect dimension usages.

commit a46542b25f
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Sep 7 18:05:04 2021 -0400

    Too much writing Julia...

commit 06f7b34d73
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Sep 3 18:43:21 2021 -0400

    Fix max pooling bug.

commit c9df713091
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Aug 21 10:44:43 2021 -0400

    First attempt at refactoring Padding, MaxPooling, and Convolution.

commit 498afa0ab7
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat Aug 7 11:15:14 2021 -0400

    Fix various bugs in the MultiLayer implementation.

commit 8105bb8583
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Aug 4 21:19:54 2021 -0400

    Refactor to use MultiLayer inside an FFN.

commit 5f7a9cd684
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Mon Aug 2 18:19:58 2021 +0200

    Filter some 'unused' layers.

commit 8a4e5b9bdf
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Aug 1 21:42:09 2021 -0400

    Refactor Highway (and fix MultiLayer).

commit 0375e7d575
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Aug 1 21:42:00 2021 -0400

    Extra paranoia to avoid include boost::visitor...

commit d3972c16da
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Aug 1 21:41:49 2021 -0400

    Just make sure boost isn't included...

commit 57884df9f6
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Aug 1 21:41:33 2021 -0400

    Split out into convenience function.

commit d136ac3df3
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Aug 1 21:41:00 2021 -0400

    Hey, this is no longer needed! :)

commit be7eeb134d
Merge: ce3c3bcee e3f4654a8
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Jul 23 16:45:08 2021 -0400

    Merge branch 'ann-vtable-attempt' into HEAD

commit e3f4654a8a
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Jul 23 16:43:46 2021 -0400

    Update RBF<> layer so tests pass.

commit ce3c3bcee6
Merge: 34cf419bb 927fabff8
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Tue Jul 20 08:56:37 2021 -0400

    Merge pull request #2 from rcurtin/ann-vtable-attempt

    Further refactoring of ANN to remove boost::visitor.

commit 927fabff8a
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Jul 13 17:53:33 2021 -0400

    Adapt the last commented test in FeedforwardNetworkTest.

commit ed8881d2b6
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Jul 13 17:48:46 2021 -0400

    Uncomment another test.

commit b05736d42c
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Jul 13 17:48:34 2021 -0400

    Make sure Parameters() returns the correct thing.

commit 522ebd11fe
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Jul 13 17:48:27 2021 -0400

    Adapt AddType<>.

commit c8c0797c9d
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Jul 13 17:19:57 2021 -0400

    Adapt a few more layers, and uncomment some more tests.

commit 2b7429594f
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Jul 13 16:47:06 2021 -0400

    Oops, I didn't really need to refactor this, but it might work.

commit ab67249f9f
Author: Ryan Curtin <ryan@ratml.org>
Date:   Thu Jul 8 17:49:57 2021 -0400

    Fix additional warnings.

commit a5bb31b824
Author: Ryan Curtin <ryan@ratml.org>
Date:   Thu Jul 8 17:38:04 2021 -0400

    Remove debugging output.

commit 188759042c
Author: Ryan Curtin <ryan@ratml.org>
Date:   Thu Jul 8 17:37:42 2021 -0400

    Fix a compilation warning.

commit 130890c85d
Author: Ryan Curtin <ryan@ratml.org>
Date:   Thu Jul 8 17:36:09 2021 -0400

    Some additional refactoring and cleanups.

    Notably, the adapted layers no longer need an input size.

commit 32b19eb7ef
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jul 7 19:08:32 2021 -0400

    Refactor Reparametrization layer.

commit d180cc3639
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jul 7 19:08:12 2021 -0400

    Serialize output dimensions also.

commit a87672665c
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jul 7 18:24:15 2021 -0400

    Remove unnecessary copy/move constructor/operators.

commit af68997a6c
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jul 7 18:22:30 2021 -0400

    This function should be const.

commit 60fe292da9
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jul 7 18:22:10 2021 -0400

    These are all the default versions anyway (but don't consider inheritance...).

commit 30e4ff741b
Author: Ryan Curtin <ryan@ratml.org>
Date:   Tue Jul 6 16:22:37 2021 -0400

    Update comments.

commit 7f7f56481f
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Jul 5 21:53:33 2021 -0400

    Remove unnecessary functions.

commit 30aca349cf
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Jul 5 19:26:55 2021 -0400

    Fix train/test modes.

commit ffba7a966c
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Jul 5 18:52:20 2021 -0400

    Remove unnecessary utilities.

commit a3dd3739c6
Author: Ryan Curtin <ryan@ratml.org>
Date:   Mon Jul 5 18:51:18 2021 -0400

    Change 'deterministic' to 'training'.

commit afa76d6ae8
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sun Jul 4 16:05:47 2021 -0400

    Make sure that we save layerOutputs.back() in case we need it later...

commit 2875194721
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jun 23 19:58:38 2021 -0400

    Fix FFNReturnModel test.

commit 292bfbef9c
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jun 23 19:32:34 2021 -0400

    Serialize deterministic in whatever state it is currently in---no assumptions.

commit a30e15fca7
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jun 23 19:32:20 2021 -0400

    Remove unnecessary output.

commit 5d27a4d43e
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jun 23 19:24:36 2021 -0400

    Add the 'MultiLayer', although maybe we can just use the FFN class itself?

commit c2b54b7af6
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jun 23 19:24:21 2021 -0400

    Set the size correctly in Predict() and fix a few other errors.

commit 0c307e89de
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Jun 18 14:11:47 2021 -0400

    Initialize totalInputSize and totalOutputSize in the right place.

commit d2fd462a8c
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Jun 18 13:51:46 2021 -0400

    Use aliases for layer outputs and deltas.

commit 064cb7b296
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri Jun 18 12:42:29 2021 -0400

    Okay, this passes FFVanillaNetworkTest!

commit 9a521c8080
Author: Ryan Curtin <ryan@ratml.org>
Date:   Wed Jun 16 14:07:23 2021 -0400

    Step 1: something compiles at all.

commit 5220de7d14
Author: Ryan Curtin <ryan@ratml.org>
Date:   Fri May 28 12:52:21 2021 -0400

    Fix some minor compilation issues.

commit f8123469e9
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat May 22 05:27:37 2021 -0400

    Add serialization file.

commit 1b3ea01b45
Author: Ryan Curtin <ryan@ratml.org>
Date:   Sat May 22 05:19:41 2021 -0400

    In-progress, does not quite compile yet.

commit 34cf419bb8
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Jan 31 19:05:00 2021 +0100

    Update FFN tests to use the base layer class.

commit b700f8d311
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Jan 31 04:36:35 2021 +0100

    Update FFN copy/move constructor tests to use the layer base class.

commit eafac8609d
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sat Jan 30 04:39:08 2021 +0100

    Add Clone() function which handles polymorphism correctly.

commit 0c4db57106
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Fri Jan 29 10:14:53 2021 +0530

    typo fix

commit 21057324f3
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Thu Jan 28 16:21:27 2021 +0530

    add ResetCell and Reward methods to base class and add method to push layer to ffn model

commit 43e7639e61
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Tue Jan 26 00:47:55 2021 +0100

    Use layer base class for the network initialization.

commit fadaaa59a2
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Mon Jan 25 00:04:26 2021 +0100

    Update layer to use updated layer base class interface.

commit 3cdd972be8
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Jan 24 23:17:26 2021 +0100

    Adjust determenistic parameter interface.

commit 5b453b712a
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Jan 24 23:14:04 2021 +0100

    Add utiliy functions to update layer parameters and states.

commit c641bc527c
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Jan 24 23:13:09 2021 +0100

    Restructure FFN class to use the layer base class.

commit 9c29e5b603
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Tue Jan 19 21:17:32 2021 +0530

    update weight_norm layer to use abstract class, and some other fixes

commit 5d62339e02
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Sun Jan 17 18:04:32 2021 +0530

    update fast_lstm to use abstract class (without unit test)

commit f2f84991ce
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Sun Jan 17 17:30:36 2021 +0530

    migrate vr_class_reward to loss_functions

commit ecd2fae814
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Sat Jan 16 10:32:18 2021 +0530

    update reinforce_normal and reparametrization layer to use abstract class

commit 826be5b404
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Fri Jan 15 23:29:21 2021 +0530

    update virtual_batch_norm to support abstract class

commit f694114805
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Fri Jan 15 23:14:22 2021 +0530

    update select, subview, padding and transposed convolution to use abstract class

commit dfec0f787d
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Fri Jan 15 18:28:43 2021 +0530

    updated sequential layer to use abstract class

commit c6675f4ba3
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Fri Jan 15 16:34:30 2021 +0530

    update concat and highway layers to use abstract class

commit 0da7330f01
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Fri Jan 15 09:11:57 2021 +0530

    update positional_encoding and multiply_merge layers to use abstract class

commit c30d9ed85c
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Fri Jan 15 07:18:52 2021 +0530

    slight style fixes and lexicographical ordering of layer_types

commit cfdb84173a
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Thu Jan 14 11:22:58 2021 +0530

    some corrections and updating layer_norm, max_pooling and mean_pooling

commit 97d865df79
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Thu Jan 14 10:25:47 2021 +0530

    correction in convolution and update join and glimpse layer to use abstract class

commit 270aab791e
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Thu Jan 14 10:05:39 2021 +0530

    update glimpse layer to use abstract class method

commit 28d56b1693
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Wed Jan 13 10:52:22 2021 +0530

    corrected documentation and updated convolution layer

commit 09e2995346
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Wed Jan 13 09:19:09 2021 +0530

    update multihead attention layer

commit dd6b70c2f2
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Tue Jan 12 09:34:23 2021 +0530

    update concatenate layer

commit 004818cca2
Author: Mrityunjay Tripathi <mrityunjay2668@gmail.com>
Date:   Mon Jan 11 23:26:42 2021 +0530

    update bilinear_interpolation, linear3d, lookup and base_layer

commit aef1158187
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Jan 3 23:31:22 2021 +0100

    Add Deterministic() to the abstract layer class.

commit 3f4d24f7f5
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Jan 3 23:06:40 2021 +0100

    Update SpatialDropout layer to use the abstract layer class and add typedef.

commit 5193102f09
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sat Jan 2 20:57:20 2021 +0100

    Update PReLU layer to use the abstract layer class and add typedef.

commit a57d13f838
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sat Jan 2 20:40:07 2021 +0100

    Update CELU layer to use the abstract layer class and add typedef.

commit 47f35337f6
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sat Jan 2 20:25:57 2021 +0100

    Update CReLU layer to use the abstract layer class and add typedef.

commit cd05cbecd4
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sat Jan 2 18:54:42 2021 +0100

    Update MultiplyConstant layer to use the abstract layer class and add typedef.

commit 8b0abc219a
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Dec 27 22:57:13 2020 +0100

    Update Softshrink layer to use the abstract layer class and add typedef.

commit 8cf621f4fd
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Dec 27 19:57:37 2020 +0100

    Update Softmin layer to use the abstract layer class and add typedef.

commit 3b0318a50d
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Dec 27 17:44:04 2020 +0100

    Update Softmax layer to use the abstract layer class and add typedef.

commit f4f66af580
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Dec 27 17:23:13 2020 +0100

    Update Constant layer to use the abstract layer class and add typedef.

commit cebce07398
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Dec 27 16:25:25 2020 +0100

    Update LeakyReLU layer to use the abstract layer class and add typedef.

commit 0217a19b25
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Dec 27 14:39:47 2020 +0100

    Update HardShrink layer to use the abstract layer class and add typedef.

commit 69f0d5b56a
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Dec 27 14:24:29 2020 +0100

    Update NoisyLinear layer to use the abstract layer class and add typedef.

commit c927971363
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sun Dec 27 13:43:57 2020 +0100

    Update LinearNoBias layer to use the abstract layer class and add typedef.

commit dea01f7215
Author: Aakash kaushik <kaushikaakash7539@gmail.com>
Date:   Fri Dec 25 22:41:46 2020 +0530

    tests pass

commit 09d905f2e5
Author: Aakash kaushik <aakash@pop-os.localdomain>
Date:   Fri Dec 25 17:28:02 2020 +0530

    update adaptive_mean_pooling_layer to use the abstract class and added typedef

commit a893867f37
Author: Aakash kaushik <aakash@pop-os.localdomain>
Date:   Fri Dec 25 17:27:30 2020 +0530

    update adaptive_mean_pooling_layer to use the abstract class and added typedef

commit 66853d1a5b
Author: Aakash kaushik <aakash@pop-os.localdomain>
Date:   Fri Dec 25 16:57:51 2020 +0530

    update adaptive_max_pooling_layer to use the abstract class and added typedef

commit 60afb1e1bc
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Sat Dec 26 01:13:17 2020 +0100

    Update Dropconnect layer to use the abstract layer class and add typedef.

commit 322753be75
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Thu Dec 24 11:57:47 2020 +0100

    Update HardTanH layer to use the abstract layer class and add typedef.

commit 80ecca1356
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Thu Dec 24 11:41:37 2020 +0100

    Update Dropout layer to use the abstract layer class and add typedef.

commit 9acac6838d
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Thu Dec 24 01:23:32 2020 +0100

    Update ELU/SELU layer to use the abstract layer class and add typedefs.

commit 5ea2b93618
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Thu Dec 24 00:59:45 2020 +0100

    Update SoftMax layer to use the abstract layer class and add typedef.

commit 2232915792
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Thu Dec 24 00:39:44 2020 +0100

    Update FlexibleReLU layer to use the abstract layer class, add documentation and typedef.

commit 4fd7ae9da7
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Thu Dec 24 00:38:25 2020 +0100

    Remove functions that are already part of the base class.

commit a6995f83ee
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Wed Dec 23 17:14:07 2020 +0100

    Update Linear layer to use the abstract Layer class and add documentation and typedef.

commit a3f746580b
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Wed Dec 23 17:12:32 2020 +0100

    Clean-up abstract layer class and add some further documentation.

commit aa6d2b1aad
Author: Marcus Edel <marcus.edel@fu-berlin.de>
Date:   Mon Dec 21 17:23:08 2020 +0100

    Swap boost::variant with vtable.
This commit is contained in:
shubham1206agra
2022-05-10 13:53:21 +05:30
parent 0023ded6a8
commit 1ea72a60ab
356 changed files with 16844 additions and 22263 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.ci/linux-steps.yaml
+1
View File
@@ -80,6 +80,7 @@ steps:
mkdir build && cd build
if [ "$(binding)" == "go" ]; then
export GOPATH=$PWD/src/mlpack/bindings/go
export GO111MODULE=off
go get -u -t gonum.org/v1/gonum/...
fi
cmake $(CMakeArgs) -DPYTHON_EXECUTABLE=`which python` -DCEREAL_INCLUDE_DIR=/usr/include/ ..
.ci/macos-steps.yaml
+1
View File
@@ -33,6 +33,7 @@ steps:
mkdir build && cd build
if [ "$(binding)" == "go" ]; then
export GOPATH=$PWD/src/mlpack/bindings/go
export GO111MODULE=off
go get -u -t gonum.org/v1/gonum/...
fi
if [ "$(binding)" == "python" ]; then
.github/workflows/main.yml
+1
View File
@@ -57,6 +57,7 @@ jobs:
run: |
remotes::install_deps(dependencies = TRUE)
remotes::install_cran("roxygen2")
remotes::install_cran("pkgbuild")
shell: Rscript {0}
- name: CMake
CMake/FindGo.cmake
+2 -2
View File
@@ -14,8 +14,8 @@ if (GO_EXECUTABLE)
RESULT_VARIABLE RESULT
)
if (RESULT EQUAL 0)
string(REGEX REPLACE ".*([0-9]+\\.[0-9]+\(\\.[0-9]+\)?).*" "\\1"
GO_VERSION_STRING ${GO_VERSION_STRING})
string(REGEX MATCH "([0-9]+\\.[0-9]+\(\\.[0-9]+\)?)"
GO_VERSION_STRING "${GO_VERSION_STRING}")
endif()
endif()
CMake/FindGonum.cmake
+6 -5
View File
@@ -4,20 +4,21 @@
if (GO_EXECUTABLE)
execute_process(
COMMAND ${GO_EXECUTABLE} list gonum.org/v1/gonum/mat
OUTPUT_VARIABLE GONUM_VERSION_STRING
OUTPUT_VARIABLE GONUM_RAW_STRING
RESULT_VARIABLE RESULT
)
if (RESULT EQUAL 0)
string(REGEX REPLACE ".*([0-9]+\\.[0-9]+\\.[0-9]+[\n]+).*" "\\1"
GONUM_VERSION_STRING ${GONUM_VERSION_STRING})
string(REGEX REPLACE "\n$" ""
GONUM_VERSION_STRING ${GONUM_VERSION_STRING})
GONUM_RAW_STRING ${GONUM_RAW_STRING})
if ("${GONUM_RAW_STRING}" STREQUAL "gonum.org/v1/gonum/mat")
set(GONUM_FOUND 1)
endif()
endif()
endif()
include(FindPackageHandleStandardArgs)
find_package_handle_standard_args(
Gonum
REQUIRED_VARS GONUM_VERSION_STRING
REQUIRED_VARS GONUM_FOUND
FAIL_MESSAGE "Gonum not found"
)
COPYRIGHT.txt
+1 -1
View File
@@ -7,7 +7,7 @@ Source:
Files: *
Copyright:
Copyright 2008-2021, Ryan Curtin <ryan@ratml.org>
Copyright 2008-2022, Ryan Curtin <ryan@ratml.org>
Copyright 2008-2013, Bill March <march@gatech.edu>
Copyright 2008-2012, Dongryeol Lee <dongryel@cc.gatech.edu>
Copyright 2008-2013, Nishant Mehta <niche@cc.gatech.edu>
HISTORY.md
+2
View File
@@ -1,5 +1,7 @@
### mlpack ?.?.?
###### ????-??-??
* Fix `Perceptron` to work with cross-validation framework (#3190).
* Migrate from boost tests to Catch2 framework (#2523), (#2584).
* Bump minimum armadillo version from 8.400 to 9.800 (#3043), (#3048).
src/mlpack/bindings/python/copy_artifacts.py
+4 -8
View File
@@ -6,18 +6,14 @@
# terms of the 3-clause BSD license. You should have received a copy of the
# 3-clause BSD license along with mlpack. If not, see
# http://www.opensource.org/licenses/BSD-3-Clause for more information.
import sys
import sysconfig
import shutil
import os
import glob
directory = 'build/lib.' + \
sysconfig.get_platform() + \
'-' + \
str(sys.version_info[0]) + \
'.' + \
str(sys.version_info[1]) + \
'/mlpack/'
# Match any lib.$platform*/mlpack/ directory.
directory = glob.glob('build/lib.' + sysconfig.get_platform() + '*/mlpack/')[0]
directory = directory.replace('\\', '/')
# Now copy all the files from the directory to the desired location.
for f in os.listdir(directory):
src/mlpack/bindings/python/is_serializable.hpp
+2 -2
View File
@@ -21,7 +21,7 @@ namespace python {
template<typename T>
inline bool IsSerializable(
util::ParamData& /* d */,
const typename boost::disable_if<data::HasSerialize<T>>::type* = 0)
const typename std::enable_if<!data::HasSerialize<T>::value>::type* = 0)
{
return false;
}
@@ -29,7 +29,7 @@ inline bool IsSerializable(
template<typename T>
inline bool IsSerializable(
util::ParamData& /* d */,
const typename boost::enable_if<data::HasSerialize<T>>::type* = 0)
const typename std::enable_if<data::HasSerialize<T>::value>::type* = 0)
{
return true;
}
src/mlpack/core/cereal/CMakeLists.txt
-2
View File
@@ -7,8 +7,6 @@ set(SOURCES
pair_associative_container.hpp
pointer_wrapper.hpp
pointer_vector_wrapper.hpp
pointer_variant_wrapper.hpp
pointer_vector_variant_wrapper.hpp
unordered_map.hpp
)
src/mlpack/core/cereal/pointer_variant_wrapper.hpp
-159
View File
@@ -1,159 +0,0 @@
/**
* @file core/cereal/pointer_variant_wrapper.hpp
* @author Omar Shrit
*
* Implementation of a boost::variant wrapper to enable the serialization of
* the pointers inside boost variant in cereal
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_CORE_CEREAL_POINTER_VARIANT_WRAPPER_HPP
#define MLPACK_CORE_CEREAL_POINTER_VARIANT_WRAPPER_HPP
#include <cereal/archives/json.hpp>
#include <cereal/archives/portable_binary.hpp>
#include <cereal/archives/xml.hpp>
#include <cereal/types/boost_variant.hpp>
#include <boost/variant.hpp>
#include <boost/variant/variant_fwd.hpp>
#include <boost/variant/static_visitor.hpp>
#include "pointer_wrapper.hpp"
namespace cereal {
// Forward declaration.
template<typename... VariantTypes>
class PointerVariantWrapper;
/**
* Serialize a boost variant in which the variant it self is a raw pointer.
* This wrapper will wrap each variant independently by encapsulating each variant
* into the PoninterWrapper we have created already.
*
* @param t A reference to boost variant that holds raw pointer.
*/
template<typename... VariantTypes>
inline PointerVariantWrapper<VariantTypes...>
make_pointer_variant(boost::variant<VariantTypes...>& t)
{
return PointerVariantWrapper<VariantTypes...>(t);
}
template<class Archive>
struct save_visitor : public boost::static_visitor<void>
{
save_visitor(Archive& ar) : ar(ar) {}
template<class T>
void operator()(const T* value) const
{
ar(CEREAL_POINTER(value));
}
template<typename... Types>
void operator()(boost::variant<Types*...>& value) const
{
ar(make_pointer_variant(value));
}
Archive& ar;
};
template<typename T>
struct load_visitor : public boost::static_visitor<void>
{
template<typename Archive, typename VariantType>
static void load_impl(Archive& ar, VariantType& variant, std::true_type)
{
// Note that T will be a pointer type.
T loadVariant;
ar(CEREAL_POINTER(loadVariant));
variant = loadVariant;
}
template<typename Archive, typename VariantType>
static void load_impl(Archive& ar, VariantType& value, std::false_type)
{
// This must be a nested boost::variant.
T loadVariant;
ar(make_pointer_variant(loadVariant));
value = loadVariant;
}
template<typename Archive, typename VariantType>
static void load(Archive& ar, VariantType& variant)
{
// Delegate to the proper load_impl() overload depending on whether T is a
// pointer type. If T is not a pointer type, then we expect it to be a
// nested boost::variant.
load_impl(ar, variant, typename std::is_pointer<T>::type());
}
};
/**
* The objective of this class is to create a wrapper for
* boost::variant.
* Cereal supports the serialization of boost::variant, but
* we need to serialize it if it holds a raw pointers.
* This class depeds on the PointerWrapper we have already created in which it is
* used to serialize each variant independently
*/
template<typename... VariantTypes>
class PointerVariantWrapper
{
public:
PointerVariantWrapper(boost::variant<VariantTypes...>& pointerVar) :
pointerVariant(pointerVar)
{}
template<class Archive>
void save(Archive& ar) const
{
// which represents the index in std::variant.
int which = pointerVariant.which();
ar(CEREAL_NVP(which));
save_visitor<Archive> s(ar);
boost::apply_visitor(s, pointerVariant);
}
template<class Archive>
void load(Archive& ar)
{
// Load the size of the serialized type.
int which;
ar(CEREAL_NVP(which));
// Create function pointers to each overload of load_visitor<T>::load, for
// all T in VariantTypes.
using LoadFuncType = void(*)(Archive&, boost::variant<VariantTypes...>&);
LoadFuncType loadFuncArray[] = { &load_visitor<VariantTypes>::load... };
if (which >= int(sizeof(loadFuncArray)/sizeof(loadFuncArray[0])))
throw std::runtime_error("Invalid 'which' selector when"
"deserializing boost::variant");
loadFuncArray[which](ar, pointerVariant);
}
private:
boost::variant<VariantTypes...>& pointerVariant;
};
/**
* Cereal does not support the serialization of raw pointer.
* This macro enable developers to serialize boost::variant that holds raw
* pointers by using the above PointerVariantWrapper class which replace the
* internal raw pointers by smart pointer internally.
*
* @param T boost::variant that holds raw pointer to be serialized.
*/
#define CEREAL_VARIANT_POINTER(T) cereal::make_pointer_variant(T)
} // namespace cereal
#endif // CEREAL_POINTER_VARIANT_WRAPPER_HPP
src/mlpack/core/cereal/pointer_vector_variant_wrapper.hpp
-97
View File
@@ -1,97 +0,0 @@
/**
* @file core/cereal/pointer_vector_variant_wrapper.hpp
* @author Omar Shrit
*
* Implementation of a boost::variant wrapper to enable the serialization of
* the pointers inside boost variant in cereal
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_CORE_CEREAL_POINTER_VECTOR_VARIANT_WRAPPER_HPP
#define MLPACK_CORE_CEREAL_POINTER_VECTOR_VARIANT_WRAPPER_HPP
#include "pointer_wrapper.hpp"
#include "pointer_variant_wrapper.hpp"
#include "pointer_vector_wrapper.hpp"
namespace cereal {
// Forward declaration
template<typename... VariantTypes>
class PointerVectorVariantWrapper;
/**
* Serialize a std::vector of boost variants in which the variant in each boost
* variant is a raw pointer.
* This wrapper will wrap each boost variant independently by encapsulating each
* boost variant into the PoninterVariantWrapper we have created already.
*
* @param t A reference to a vector of boost variants that holds raw pointer.
*/
template<typename... VariantTypes>
inline PointerVectorVariantWrapper<VariantTypes...>
make_vector_pointer_variant(std::vector<boost::variant<VariantTypes...>>& t)
{
return PointerVectorVariantWrapper<VariantTypes...>(t);
}
/**
* The objective of this class is to create a wrapper for
* a vector of boost::variant that holds pointer.
* Cereal supports the serialization of boost::variant, but
* we need to serialize it if it holds a vector of boost::variant that holds a
* pointers.
*/
template<typename... VariantTypes>
class PointerVectorVariantWrapper
{
public:
PointerVectorVariantWrapper(
std::vector<boost::variant<VariantTypes...>>& vecPointerVar)
: vectorPointerVariant(vecPointerVar)
{}
template<class Archive>
void save(Archive& ar) const
{
size_t vecSize = vectorPointerVariant.size();
ar(CEREAL_NVP(vecSize));
for (size_t i = 0; i < vectorPointerVariant.size(); ++i)
{
ar(CEREAL_VARIANT_POINTER(vectorPointerVariant.at(i)));
}
}
template<class Archive>
void load(Archive& ar)
{
size_t vecSize = 0;
ar(CEREAL_NVP(vecSize));
vectorPointerVariant.resize(vecSize);
for (size_t i = 0; i < vectorPointerVariant.size(); ++i)
{
ar(CEREAL_VARIANT_POINTER(vectorPointerVariant.at(i)));
}
}
private:
std::vector<boost::variant<VariantTypes...>>& vectorPointerVariant;
};
/**
* Cereal does not support the serialization of raw pointer.
* This macro enable developers to serialize a std vector that holds boost::variants
* that holds raw pointers by using the above PointerVectorVariantWrapper class
* which replace the internal raw pointers by smart pointer internally.
*
* @param T std::vector<boost::variant> that holds raw pointer to be serialized.
*/
#define CEREAL_VECTOR_VARIANT_POINTER(T) cereal::make_vector_pointer_variant(T)
} // namespace cereal
#endif // CEREAL_POINTER_VECTOR_VARIANT_WRAPPER_HPP
src/mlpack/core/util/size_checks.hpp
+14 -5
View File
@@ -26,18 +26,27 @@ namespace util {
* error generation.
* @param addInfo Name to use for labels for precise error generation. Default
* is "labels"; for example, "weights" could also be used.
* @param isDataTranspose Bool parameter which can be set true to transpose data
* before size-check. Default is false.
* @param isLabelTranspose Bool parameter which can be set true to transpose label
* before size-check. Default is false.
*/
template<typename DataType, typename LabelsType>
inline void CheckSameSizes(const DataType& data,
const LabelsType& label,
const std::string& callerDescription,
const std::string& addInfo = "labels")
{
if (data.n_cols != label.n_cols)
const std::string& addInfo = "labels",
const bool& isDataTranspose = false,
const bool& isLabelTranspose = false)
{
const size_t dataPoints = (isDataTranspose == true) ? data.n_rows : data.n_cols;
const size_t labelPoints = (isLabelTranspose == true) ? label.n_rows : label.n_cols;
if (dataPoints != labelPoints)
{
std::ostringstream oss;
oss << callerDescription << ": number of points (" << data.n_cols << ") "
<< "does not match number of " << addInfo << " (" << label.n_cols
oss << callerDescription << ": number of points (" << dataPoints << ") "
<< "does not match number of " << addInfo << " (" << labelPoints
<< ")!" << std::endl;
throw std::invalid_argument(oss.str());
}
src/mlpack/methods/ann/CMakeLists.txt
+2 -9
View File
@@ -3,24 +3,17 @@
set(SOURCES
ffn.hpp
ffn_impl.hpp
forward_decls.hpp
make_alias.hpp
rnn.hpp
rnn_impl.hpp
brnn.hpp
brnn_impl.hpp
layer_names.hpp
)
add_subdirectory(visitor)
add_subdirectory(activation_functions)
add_subdirectory(init_rules)
add_subdirectory(layer)
add_subdirectory(loss_functions)
add_subdirectory(convolution_rules)
add_subdirectory(gan)
add_subdirectory(rbm)
add_subdirectory(augmented)
add_subdirectory(regularizer)
add_subdirectory(util)
# Add directory name to sources.
set(DIR_SRCS)
src/mlpack/methods/ann/convolution_rules/naive_convolution.hpp
+21 -30
View File
@@ -57,9 +57,15 @@ class NaiveConvolution
const size_t dilationW = 1,
const size_t dilationH = 1)
{
output = arma::zeros<arma::Mat<eT> >(
(input.n_rows - (filter.n_rows - 1) * dilationW - 1) / dW + 1,
(input.n_cols - (filter.n_cols - 1) * dilationH - 1) / dH + 1);
// Compute the output size. The filterRows and filterCols computation must
// take into account the fact that dilation only adds rows or columns
// *between* filter elements. So, e.g., a dilation of 2 on a kernel size of
// 3x3 means an effective kernel size of 5x5, *not* 6x6.
const size_t filterRows = filter.n_rows * dilationH - (dilationH - 1);
const size_t filterCols = filter.n_cols * dilationW - (dilationW - 1);
const size_t outputRows = (input.n_rows - filterRows + dH) / dH;
const size_t outputCols = (input.n_cols - filterCols + dW) / dW;
output.zeros(outputRows, outputCols);
// It seems to be about 3.5 times faster to use pointers instead of
// filter(ki, kj) * input(leftInput + ki, topInput + kj) and output(i, j).
@@ -103,37 +109,22 @@ class NaiveConvolution
const size_t dilationW = 1,
const size_t dilationH = 1)
{
size_t outputRows = (input.n_rows - 1) * dW + 2 * (filter.n_rows - 1)
* dilationW + 1;
size_t outputCols = (input.n_cols - 1) * dH + 2 * (filter.n_cols - 1)
* dilationH + 1;
for (size_t i = 0; i < dW; ++i)
{
if (((((i + outputRows - 2 * (filter.n_rows - 1) * dilationW - 1) % dW)
+ dW) % dW) == i){
outputRows += i;
break;
}
}
for (size_t i = 0; i < dH; ++i)
{
if (((((i + outputCols - 2 * (filter.n_cols - 1) * dilationH - 1) % dH)
+ dH) % dH) == i){
outputCols += i;
break;
}
}
// First, compute the necessary padding for the full convolution. It is
// possible that this might be an overestimate. Note that these variables
// only hold the padding on one side of the input.
const size_t filterRows = filter.n_rows * dilationH - (dilationH - 1);
const size_t filterCols = filter.n_cols * dilationW - (dilationW - 1);
const size_t paddingRows = filterRows - 1;
const size_t paddingCols = filterCols - 1;
// Pad filter and input to the working output shape.
arma::Mat<eT> inputPadded = arma::zeros<arma::Mat<eT> >(outputRows,
outputCols);
inputPadded.submat((filter.n_rows - 1) * dilationW, (filter.n_cols - 1)
* dilationH, (filter.n_rows - 1) * dilationW + input.n_rows - 1,
(filter.n_cols - 1) * dilationH + input.n_cols - 1) = input;
arma::Mat<eT> inputPadded(input.n_rows + 2 * paddingRows,
input.n_cols + 2 * paddingCols, arma::fill::zeros);
inputPadded.submat(paddingRows, paddingCols, paddingRows + input.n_rows - 1,
paddingCols + input.n_cols - 1) = input;
NaiveConvolution<ValidConvolution>::Convolution(inputPadded, filter,
output, 1, 1, dilationW, dilationH);
output, dW, dH, dilationW, dilationH);
}
/*
src/mlpack/methods/ann/ffn.hpp
+359 -327
View File
@@ -15,46 +15,49 @@
#include <mlpack/prereqs.hpp>
#include "visitor/delete_visitor.hpp"
#include "visitor/delta_visitor.hpp"
#include "visitor/output_height_visitor.hpp"
#include "visitor/output_parameter_visitor.hpp"
#include "visitor/output_width_visitor.hpp"
#include "visitor/reset_visitor.hpp"
#include "visitor/weight_size_visitor.hpp"
#include "visitor/copy_visitor.hpp"
#include "visitor/loss_visitor.hpp"
#include "forward_decls.hpp"
#include "init_rules/network_init.hpp"
#include <mlpack/methods/ann/layer/layer_types.hpp>
#include <mlpack/methods/ann/layer/layer.hpp>
#include <mlpack/methods/ann/layer/multi_layer.hpp>
#include <mlpack/methods/ann/init_rules/random_init.hpp>
#include <mlpack/methods/ann/layer/layer_traits.hpp>
#include <mlpack/methods/ann/loss_functions/negative_log_likelihood.hpp>
#include <ensmallen.hpp>
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
/**
* Implementation of a standard feed forward network.
* Implementation of a standard feed forward network. Any layer that inherits
* from the base `Layer` class can be added to this model. For recursive neural
* networks, see the `RNN` class.
*
* In general, a network can be created by using the `Add()` method to add
* layers to the network. Then, training can be performed with `Train()`, and
* data points can be passed through the trained network with `Predict()`.
*
* Although the actual types passed as input will be matrix objects with one
* data point per column, each data point can be a tensor of arbitrary shape.
* If data points are not 1-dimensional vectors, then set the shape of the input
* with `InputDimensions()` before calling `Train()`.
*
* More granular functionality is available with `Forward()`, Backward()`, and
* `Evaluate()`, or even by accessing the individual layers directly with
* `Network()`.
*
* @tparam OutputLayerType The output layer type used to evaluate the network.
* @tparam InitializationRuleType Rule used to initialize the weight matrix.
* @tparam CustomLayers Any set of custom layers that could be a part of the
* feed forward network.
* @tparam MatType Type of matrix to be given as input to the network.
* @tparam MatType Type of matrix to be produced as output from the last
* layer.
*/
template<
typename OutputLayerType = NegativeLogLikelihood<>,
typename InitializationRuleType = RandomInitialization,
typename... CustomLayers
>
typename OutputLayerType = NegativeLogLikelihood,
typename InitializationRuleType = RandomInitialization,
typename MatType = arma::mat>
class FFN
{
public:
//! Convenience typedef for the internal model construction.
using NetworkType = FFN<OutputLayerType, InitializationRuleType>;
/**
* Create the FFN object.
*
@@ -72,56 +75,73 @@ class FFN
InitializationRuleType initializeRule = InitializationRuleType());
//! Copy constructor.
FFN(const FFN&);
FFN(const FFN& other);
//! Move constructor.
FFN(FFN&&);
//! Copy/move assignment operator.
FFN& operator = (FFN);
//! Destructor to release allocated memory.
~FFN();
FFN(FFN&& other);
//! Copy operator.
FFN& operator=(const FFN& other);
//! Move assignment operator.
FFN& operator=(FFN&& other);
/**
* Check if the optimizer has MaxIterations() parameter, if it does
* then check if it's value is less than the number of datapoints
* in the dataset.
* Add a new layer to the model.
*
* @tparam OptimizerType Type of optimizer to use to train the model.
* @param optimizer optimizer used in the training process.
* @param samples Number of datapoints in the dataset.
* @param args The layer parameter.
*/
template<typename OptimizerType>
typename std::enable_if<
HasMaxIterations<OptimizerType, size_t&(OptimizerType::*)()>
::value, void>::type
WarnMessageMaxIterations(OptimizerType& optimizer, size_t samples) const;
template <typename LayerType, typename... Args>
void Add(Args... args)
{
network.template Add<LayerType>(args...);
inputDimensionsAreSet = false;
}
/**
* Check if the optimizer has MaxIterations() parameter, if it
* doesn't then simply return from the function.
* Add a new layer to the model. Note that any trainable weights of this
* layer will be reset! (Any constant parameters are kept.)
*
* @tparam OptimizerType Type of optimizer to use to train the model.
* @param optimizer optimizer used in the training process.
* @param samples Number of datapoints in the dataset.
* @param layer The Layer to be added to the model.
*/
template<typename OptimizerType>
typename std::enable_if<
!HasMaxIterations<OptimizerType, size_t&(OptimizerType::*)()>
::value, void>::type
WarnMessageMaxIterations(OptimizerType& optimizer, size_t samples) const;
void Add(Layer<MatType>* layer)
{
network.Add(layer);
inputDimensionsAreSet = false;
}
//! Get the layers of the network.
const std::vector<Layer<MatType>*>& Network() const
{
return network.Network();
}
/**
* Modify the network model. Be careful! If you change the structure of the
* network or parameters for layers, its state may become invalid, and the
* next time it is used for any operation the parameters will be reset.
*
* Don't add any layers like this; use `Add()` instead.
*/
std::vector<Layer<MatType>*>& Network()
{
// We can no longer make any assumptions... the user may change anything.
inputDimensionsAreSet = false;
layerMemoryIsSet = false;
return network.Network();
}
/**
* Train the feedforward network on the given input data using the given
* optimizer.
*
* This will use the existing model parameters as a starting point for the
* optimization. If this is not what you want, then you should access the
* parameters vector directly with Parameters() and modify it as desired.
* If no parameters have ever been set (e.g. if `Parameters()` is an empty
* matrix), or if the parameters' size does not match the number of weights
* needed for the current input size (as given by `predictors` and optionally
* set further by `InputDimensions()`), then the network will be initialized
* using `InitializeRuleType`.
*
* If you want to pass in a parameter and discard the original parameter
* object, be sure to use std::move to avoid unnecessary copy.
* If parameters are the right size for the given `predictors` and
* `InputDimensions()`, then the existing parameters will be used as a
* starting point. (If you want to reinitialize, first call `Reset()`.)
*
* @tparam OptimizerType Type of optimizer to use to train the model.
* @tparam CallbackTypes Types of Callback Functions.
@@ -133,22 +153,25 @@ class FFN
* @return The final objective of the trained model (NaN or Inf on error).
*/
template<typename OptimizerType, typename... CallbackTypes>
double Train(arma::mat predictors,
arma::mat responses,
OptimizerType& optimizer,
CallbackTypes&&... callbacks);
typename MatType::elem_type Train(MatType predictors,
MatType responses,
OptimizerType& optimizer,
CallbackTypes&&... callbacks);
/**
* Train the feedforward network on the given input data. By default, the
* RMSProp optimization algorithm is used, but others can be specified
* (such as ens::SGD).
*
* This will use the existing model parameters as a starting point for the
* optimization. If this is not what you want, then you should access the
* parameters vector directly with Parameters() and modify it as desired.
* If no parameters have ever been set (e.g. if `Parameters()` is an empty
* matrix), or if the parameters' size does not match the number of weights
* needed for the current input size (as given by `predictors` and optionally
* set further by `InputDimensions()`), then the network will be initialized
* using `InitializeRuleType`.
*
* If you want to pass in a parameter and discard the original parameter
* object, be sure to use std::move to avoid unnecessary copy.
* If parameters are the right size for the given `predictors` and
* `InputDimensions()`, then the existing parameters will be used as a
* starting point. (If you want to reinitialize, first call `Reset()`.)
*
* @tparam OptimizerType Type of optimizer to use to train the model.
* @param predictors Input training variables.
@@ -159,22 +182,123 @@ class FFN
* @return The final objective of the trained model (NaN or Inf on error).
*/
template<typename OptimizerType = ens::RMSProp, typename... CallbackTypes>
double Train(arma::mat predictors,
arma::mat responses,
CallbackTypes&&... callbacks);
typename MatType::elem_type Train(MatType predictors,
MatType responses,
CallbackTypes&&... callbacks);
/**
* Predict the responses to a given set of predictors. The responses will
* reflect the output of the given output layer as returned by the
* output layer function.
*
* If you want to pass in a parameter and discard the original parameter
* object, be sure to use std::move to avoid unnecessary copy.
* Predict the responses to a given set of predictors. The responses will be
* the output of the output layer when `predictors` is passed through the
* whole network (`OutputLayerType`).
*
* @param predictors Input predictors.
* @param results Matrix to put output predictions of responses into.
* @param batchSize Batch size to use for prediction.
*/
void Predict(arma::mat predictors, arma::mat& results);
void Predict(MatType predictors,
MatType& results,
const size_t batchSize = 128);
// Return the number of weights in the model.
size_t WeightSize();
/**
* Set the logical dimensions of the input. `Train()` and `Predict()` expect
* data to be passed such that one point corresponds to one column, but this
* data is allowed to be an arbitrary higher-order tensor.
*
* So, if the input is meant to be 28x28x3 images, then the
* input data to `Train()` or `Predict()` should have 28*28*3 = 2352 rows, and
* `InputDimensions()` should be set to `{ 28, 28, 3 }`. Then, the layers of
* the network will interpret each input point as a 3-dimensional image
* instead of a 1-dimensional vector.
*
* If `InputDimensions()` is left unset before training, the data will be
* assumed to be a 1-dimensional vector.
*/
std::vector<size_t>& InputDimensions()
{
// The user may change the input dimensions, so we will have to propagate
// these changes to the network.
inputDimensionsAreSet = false;
return inputDimensions;
}
//! Get the logical dimensions of the input.
const std::vector<size_t>& InputDimensions() const { return inputDimensions; }
//! Return the current set of weights. These are linearized: this contains
//! the weights of every layer.
const MatType& Parameters() const { return parameters; }
//! Modify the current set of weights. These are linearized: this contains
//! the weights of every layer. Be careful! If you change the shape of
//! `parameters` to something incorrect, it may be re-initialized the next
//! time a forward pass is done.
MatType& Parameters() { return parameters; }
/**
* Reset the stored data of the network entirely. This resets all weights of
* each layer using `InitializationRuleType`, and prepares the network to
* accept a (flat 1-d) input size of `inputDimensionality` (if passed), or
* whatever input size has been set with `InputDimensions()`.
*
* This also resets the mode of the network to prediction mode (not training
* mode). See `SetNetworkMode()` for more information.
*/
void Reset(const size_t inputDimensionality = 0);
/**
* Set all the layers in the network to training mode, if `training` is
* `true`, or set all the layers in the network to testing mode, if `training`
* is `false`.
*/
void SetNetworkMode(const bool training);
/**
* Perform a manual forward pass of the data.
*
* `Forward()` and `Backward()` should be used as a pair, and they are
* designed mainly for advanced users. You should try to use `Predict()` and
* `Train()`, if you can.
*
* @param inputs The input data.
* @param results The predicted results.
*/
void Forward(const MatType& inputs, MatType& results);
/**
* Perform a manual partial forward pass of the data.
*
* This function is meant for the cases when users require a forward pass only
* through certain layers and not the entire network. `Forward()` and
* `Backward()` should be used as a pair, and they are designed mainly for
* advanced users. You should try to use `Predict()` and `Train()`, if you
* can.
*
* @param inputs The input data for the specified first layer.
* @param results The predicted results from the specified last layer.
* @param begin The index of the first layer.
* @param end The index of the last layer.
*/
void Forward(const MatType& inputs,
MatType& results,
const size_t begin,
const size_t end);
/**
* Perform a manual backward pass of the data.
*
* `Forward()` and `Backward()` should be used as a pair, and they are
* designed mainly for advanced users. You should try to use `Predict()` and
* `Train()` instead, if you can.
*
* @param inputs Inputs of current pass.
* @param targets The training target.
* @param gradients Computed gradients.
* @return Training error of the current pass.
*/
typename MatType::elem_type Backward(const MatType& inputs,
const MatType& targets,
MatType& gradients);
/**
* Evaluate the feedforward network with the given predictors and responses.
@@ -183,41 +307,38 @@ class FFN
* @param predictors Input variables.
* @param responses Target outputs for input variables.
*/
template<typename PredictorsType, typename ResponsesType>
double Evaluate(const PredictorsType& predictors,
const ResponsesType& responses);
typename MatType::elem_type Evaluate(const MatType& predictors,
const MatType& responses);
//! Serialize the model.
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
//
// Only ensmallen utility functions for training are found below here.
// They aren't generally useful otherwise.
//
/**
* Evaluate the feedforward network with the given parameters. This function
* is usually called by the optimizer to train the model.
* Note: this function is implemented so that it can be used by ensmallen's
* optimizers. It's not generally meant to be used otherwise.
*
* Evaluate the feedforward network with the given parameters.
*
* @param parameters Matrix model parameters.
*/
double Evaluate(const arma::mat& parameters);
typename MatType::elem_type Evaluate(const MatType& parameters);
/**
/**
* Note: this function is implemented so that it can be used by ensmallen's
* optimizers. It's not generally meant to be used otherwise.
*
* Evaluate the feedforward network with the given parameters, but using only
* a number of data points. This is useful for optimizers such as SGD, which
* require a separable objective function.
*
* @param parameters Matrix model parameters.
* @param begin Index of the starting point to use for objective function
* evaluation.
* @param batchSize Number of points to be passed at a time to use for
* objective function evaluation.
* @param deterministic Whether or not to train or test the model. Note some
* layer act differently in training or testing mode.
*/
double Evaluate(const arma::mat& parameters,
const size_t begin,
const size_t batchSize,
const bool deterministic);
/**
* Evaluate the feedforward network with the given parameters, but using only
* a number of data points. This is useful for optimizers such as SGD, which
* require a separable objective function. This just calls the overload of
* Evaluate() with deterministic = true.
* Note that the network may return different results depending on the mode it
* is in (see `SetNetworkMode()`).
*
* @param parameters Matrix model parameters.
* @param begin Index of the starting point to use for objective function
@@ -225,11 +346,14 @@ class FFN
* @param batchSize Number of points to be passed at a time to use for
* objective function evaluation.
*/
double Evaluate(const arma::mat& parameters,
const size_t begin,
const size_t batchSize);
typename MatType::elem_type Evaluate(const MatType& parameters,
const size_t begin,
const size_t batchSize);
/**
* Note: this function is implemented so that it can be used by ensmallen's
* optimizers. It's not generally meant to be used otherwise.
*
* Evaluate the feedforward network with the given parameters.
* This function is usually called by the optimizer to train the model.
* This just calls the overload of EvaluateWithGradient() with batchSize = 1.
@@ -237,10 +361,13 @@ class FFN
* @param parameters Matrix model parameters.
* @param gradient Matrix to output gradient into.
*/
template<typename GradType>
double EvaluateWithGradient(const arma::mat& parameters, GradType& gradient);
typename MatType::elem_type EvaluateWithGradient(const MatType& parameters,
MatType& gradient);
/**
/**
* Note: this function is implemented so that it can be used by ensmallen's
* optimizers. It's not generally meant to be used otherwise.
*
* Evaluate the feedforward network with the given parameters, but using only
* a number of data points. This is useful for optimizers such as SGD, which
* require a separable objective function.
@@ -252,13 +379,15 @@ class FFN
* @param batchSize Number of points to be passed at a time to use for
* objective function evaluation.
*/
template<typename GradType>
double EvaluateWithGradient(const arma::mat& parameters,
const size_t begin,
GradType& gradient,
const size_t batchSize);
typename MatType::elem_type EvaluateWithGradient(const MatType& parameters,
const size_t begin,
MatType& gradient,
const size_t batchSize);
/**
* Note: this function is implemented so that it can be used by ensmallen's
* optimizers. It's not generally meant to be used otherwise.
*
* Evaluate the gradient of the feedforward network with the given parameters,
* and with respect to only a number of points in the dataset. This is useful
* for optimizers such as SGD, which require a separable objective function.
@@ -270,253 +399,156 @@ class FFN
* @param batchSize Number of points to be processed as a batch for objective
* function gradient evaluation.
*/
void Gradient(const arma::mat& parameters,
void Gradient(const MatType& parameters,
const size_t begin,
arma::mat& gradient,
MatType& gradient,
const size_t batchSize);
/**
* Shuffle the order of function visitation. This may be called by the
* optimizer.
* Note: this function is implemented so that it can be used by ensmallen's
* optimizers. It's not generally meant to be used otherwise.
*
* Return the number of separable functions (the number of predictor points).
*/
size_t NumFunctions() const { return responses.n_cols; }
/**
* Note: this function is implemented so that it can be used by ensmallen's
* optimizers. It's not generally meant to be used otherwise.
*
* Shuffle the order of function visitation. (This is equivalent to shuffling
* the dataset during training.)
*/
void Shuffle();
/*
* Add a new module to the model.
*
* @param args The layer parameter.
*/
template <class LayerType, class... Args>
void Add(Args... args) { network.push_back(new LayerType(args...)); }
/*
* Add a new module to the model.
*
* @param layer The Layer to be added to the model.
*/
void Add(LayerTypes<CustomLayers...> layer) { network.push_back(layer); }
//! Get the network model.
const std::vector<LayerTypes<CustomLayers...> >& Model() const
{
return network;
}
//! Modify the network model. Be careful! If you change the structure of the
//! network or parameters for layers, its state may become invalid, so be sure
//! to call ResetParameters() afterwards.
std::vector<LayerTypes<CustomLayers...> >& Model() { return network; }
//! Return the number of separable functions (the number of predictor points).
size_t NumFunctions() const { return numFunctions; }
//! Return the initial point for the optimization.
const arma::mat& Parameters() const { return parameter; }
//! Modify the initial point for the optimization.
arma::mat& Parameters() { return parameter; }
//! Get the matrix of responses to the input data points.
const arma::mat& Responses() const { return responses; }
//! Modify the matrix of responses to the input data points.
arma::mat& Responses() { return responses; }
//! Get the matrix of data points (predictors).
const arma::mat& Predictors() const { return predictors; }
//! Modify the matrix of data points (predictors).
arma::mat& Predictors() { return predictors; }
/**
* Reset the module infomration (weights/parameters).
*/
void ResetParameters();
//! Serialize the model.
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
/**
* Perform the forward pass of the data in real batch mode.
* Prepare the network for training on the given data.
*
* Forward and Backward should be used as a pair, and they are designed mainly
* for advanced users. User should try to use Predict and Train unless those
* two functions can't satisfy some special requirements.
*
* @param inputs The input data.
* @param results The predicted results.
*/
template<typename PredictorsType, typename ResponsesType>
void Forward(const PredictorsType& inputs, ResponsesType& results);
/**
* Perform a partial forward pass of the data.
*
* This function is meant for the cases when users require a forward pass only
* through certain layers and not the entire network.
*
* @param inputs The input data for the specified first layer.
* @param results The predicted results from the specified last layer.
* @param begin The index of the first layer.
* @param end The index of the last layer.
*/
template<typename PredictorsType, typename ResponsesType>
void Forward(const PredictorsType& inputs ,
ResponsesType& results,
const size_t begin,
const size_t end);
/**
* Perform the backward pass of the data in real batch mode.
*
* Forward and Backward should be used as a pair, and they are designed mainly
* for advanced users. User should try to use Predict and Train unless those
* two functions can't satisfy some special requirements.
*
* @param inputs Inputs of current pass.
* @param targets The training target.
* @param gradients Computed gradients.
* @return Training error of the current pass.
*/
template<typename PredictorsType,
typename TargetsType,
typename GradientsType>
double Backward(const PredictorsType& inputs,
const TargetsType& targets,
GradientsType& gradients);
private:
// Helper functions.
/**
* The Forward algorithm (part of the Forward-Backward algorithm). Computes
* forward probabilities for each module.
*
* @param input Data sequence to compute probabilities for.
*/
template<typename InputType>
void Forward(const InputType& input);
/**
* Prepare the network for the given data.
* This function won't actually trigger training process.
* This function won't actually trigger the training process, and is
* generally only useful internally.
*
* @param predictors Input data variables.
* @param responses Outputs results from input data variables.
*/
void ResetData(arma::mat predictors, arma::mat responses);
void ResetData(MatType predictors, MatType responses);
private:
// Helper functions.
//! Use the InitializationPolicy to initialize all the weights in the network.
void InitializeWeights();
//! Make the memory of each layer point to the right place, by calling
//! SetWeightPtr() on each layer.
void SetLayerMemory();
/**
* The Backward algorithm (part of the Forward-Backward algorithm). Computes
* backward pass for module.
*/
void Backward();
/**
* Iterate through all layer modules and update the the gradient using the
* layer defined optimizer.
*/
template<typename InputType>
void Gradient(const InputType& input);
/**
* Reset the module status by setting the current deterministic parameter
* for all modules that implement the Deterministic function.
*/
void ResetDeterministic();
/**
* Reset the gradient for all modules that implement the Gradient function.
*/
void ResetGradients(arma::mat& gradient);
/**
* Swap the content of this network with given network.
* Ensure that all the locally-cached information about the network is valid,
* all parameter memory is initialized, and we can make forward and backward
* passes.
*
* @param network Desired source network.
* @param functionName Name of function to use if an exception is thrown.
* @param inputDimensionality Given dimensionality of the input data.
* @param setMode If true, the mode of the network will be set to the
* parameter given in `training`. Otherwise the mode of the network is
* left unmodified.
* @param training Mode to set the network to; `true` indicates the network
* should be set to training mode; `false` indicates testing mode.
*/
void Swap(FFN& network);
void CheckNetwork(const std::string& functionName,
const size_t inputDimensionality,
const bool setMode = false,
const bool training = false);
//! Instantiated outputlayer used to evaluate the network.
/**
* Set the input and output dimensions of each layer in the network correctly.
* The size of the input is taken, in case `inputDimensions` has not been set
* otherwise (e.g. via `InputDimensions()`). If `InputDimensions()` is not
* empty, then `inputDimensionality` is ignored.
*/
void UpdateDimensions(const std::string& functionName,
const size_t inputDimensionality = 0);
/**
* Check if the optimizer has MaxIterations() parameter, if it does then check
* if its value is less than the number of datapoints in the dataset.
*
* @tparam OptimizerType Type of optimizer to use to train the model.
* @param optimizer optimizer used in the training process.
* @param samples Number of datapoints in the dataset.
*/
template<typename OptimizerType>
typename std::enable_if<
ens::traits::HasMaxIterationsSignature<OptimizerType>::value, void
>::type
WarnMessageMaxIterations(OptimizerType& optimizer, size_t samples) const;
/**
* Check if the optimizer has MaxIterations() parameter; if it doesn't then
* simply return from the function.
*
* @tparam OptimizerType Type of optimizer to use to train the model.
* @param optimizer optimizer used in the training process.
* @param samples Number of datapoints in the dataset.
*/
template<typename OptimizerType>
typename std::enable_if<
!ens::traits::HasMaxIterationsSignature<OptimizerType>::value, void
>::type
WarnMessageMaxIterations(OptimizerType& optimizer, size_t samples) const;
//! Instantiated output layer used to evaluate the network.
OutputLayerType outputLayer;
//! Instantiated InitializationRule object for initializing the network
//! parameter.
InitializationRuleType initializeRule;
//! The input width.
size_t width;
//! All of the network is stored inside this multilayer.
MultiLayer<MatType> network;
//! The input height.
size_t height;
/**
* Matrix of (trainable) parameters. Each weight here corresponds to a layer,
* and each layer's `parameters` member is an alias pointing to parameters in
* this matrix.
*
* Note: although each layer may have its own MatType and MatType,
* ensmallen optimization requires everything to be stored in one matrix
* object, so we have chosen MatType. This could be made more flexible
* with a "wrapper" class implementing the Armadillo API.
*/
MatType parameters;
//! Indicator if we already trained the model.
bool reset;
//! Dimensions of input data.
std::vector<size_t> inputDimensions;
//! Locally-stored model modules.
std::vector<LayerTypes<CustomLayers...> > network;
//! The matrix of data points (predictors). This member is empty, except
//! during training---we must store a local copy of the training data since
//! the ensmallen optimizer will not provide training data.
MatType predictors;
//! The matrix of data points (predictors).
arma::mat predictors;
//! The matrix of responses to the input data points. This member is empty,
//! except during training.
MatType responses;
//! The matrix of responses to the input data points.
arma::mat responses;
//! Locally-stored output of the network from a forward pass; used by the
//! backward pass.
MatType networkOutput;
//! Locally-stored output of the backward pass; used by the gradient pass.
MatType networkDelta;
//! Locally-stored error of the backward pass; used by the gradient pass.
MatType error;
//! Matrix of (trained) parameters.
arma::mat parameter;
//! If true, each layer has its memory properly set for a forward/backward
//! pass.
bool layerMemoryIsSet;
//! The number of separable functions (the number of predictor points).
size_t numFunctions;
//! If true, each layer has its inputDimensions properly set, and
//! `totalInputSize` and `totalOutputSize` are valid.
bool inputDimensionsAreSet;
//! The current error for the backward pass.
arma::mat error;
//! Locally-stored delta visitor.
DeltaVisitor deltaVisitor;
//! Locally-stored output parameter visitor.
OutputParameterVisitor outputParameterVisitor;
//! Locally-stored weight size visitor.
WeightSizeVisitor weightSizeVisitor;
//! Locally-stored output width visitor.
OutputWidthVisitor outputWidthVisitor;
//! Locally-stored output height visitor.
OutputHeightVisitor outputHeightVisitor;
//! Locally-stored loss visitor
LossVisitor lossVisitor;
//! Locally-stored reset visitor.
ResetVisitor resetVisitor;
//! Locally-stored delete visitor.
DeleteVisitor deleteVisitor;
//! The current evaluation mode (training or testing).
bool deterministic;
//! Locally-stored delta object.
arma::mat delta;
//! Locally-stored input parameter object.
arma::mat inputParameter;
//! Locally-stored output parameter object.
arma::mat outputParameter;
//! Locally-stored gradient parameter.
arma::mat gradient;
//! Locally-stored copy visitor
CopyVisitor<CustomLayers...> copyVisitor;
// The GAN class should have access to internal members.
template<
typename Model,
typename InitializerType,
typename NoiseType,
typename PolicyType
>
friend class GAN;
// RNN will call `CheckNetwork()`, which is private.
friend class RNN<OutputLayerType, InitializationRuleType, MatType>;
}; // class FFN
} // namespace ann
src/mlpack/methods/ann/ffn_impl.hpp
+597 -565
View File
File diff suppressed because it is too large Load Diff
src/mlpack/methods/ann/forward_decls.hpp
+29
View File
@@ -0,0 +1,29 @@
/**
* @file forward_decls.hpp
* @author Ryan Curtin
*
* Forward declarations of network types. This is needed for some `friend`
* functionality.
*/
#ifndef MLPACK_METHODS_ANN_FORWARD_DECLS_HPP
#define MLPACK_METHODS_ANN_FORWARD_DECLS_HPP
namespace mlpack {
namespace ann {
// See ffn.hpp.
template<typename OutputLayerType,
typename InitializationRuleType,
typename MatType>
class FFN;
// See rnn.hpp.
template<typename OutputLayerType,
typename InitializationRuleType,
typename MatType>
class RNN;
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/init_rules/const_init.hpp
+7 -1
View File
@@ -98,7 +98,13 @@ class ConstInitialization
//! Get the initialization value.
double const& InitValue() const { return initVal; }
//! Modify the initialization value.
double& initValue() { return initVal; }
double& InitValue() { return initVal; }
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */)
{
ar(CEREAL_NVP(initVal));
}
private:
//! Value to be initialized with
src/mlpack/methods/ann/init_rules/glorot_init.hpp
+12 -6
View File
@@ -104,13 +104,19 @@ class GlorotInitializationType
*/
template<typename eT>
void Initialize(arma::Cube<eT>& W);
/**
* Serialize the initialization. (Nothing to serialize for this one.)
*/
template<typename Archive>
void serialize(Archive& /* ar */, const uint32_t /* version */) { }
}; // class GlorotInitializationType
template <>
template<>
template<typename eT>
inline void GlorotInitializationType<false>::Initialize(arma::Mat<eT>& W,
const size_t rows,
const size_t cols)
const size_t rows,
const size_t cols)
{
if (W.is_empty())
W.set_size(rows, cols);
@@ -120,7 +126,7 @@ inline void GlorotInitializationType<false>::Initialize(arma::Mat<eT>& W,
normalInit.Initialize(W, rows, cols);
}
template <>
template<>
template<typename eT>
inline void GlorotInitializationType<false>::Initialize(arma::Mat<eT>& W)
{
@@ -132,7 +138,7 @@ inline void GlorotInitializationType<false>::Initialize(arma::Mat<eT>& W)
normalInit.Initialize(W);
}
template <>
template<>
template<typename eT>
inline void GlorotInitializationType<true>::Initialize(arma::Mat<eT>& W,
const size_t rows,
@@ -147,7 +153,7 @@ inline void GlorotInitializationType<true>::Initialize(arma::Mat<eT>& W,
randomInit.Initialize(W, rows, cols);
}
template <>
template<>
template<typename eT>
inline void GlorotInitializationType<true>::Initialize(arma::Mat<eT>& W)
{
src/mlpack/methods/ann/init_rules/he_init.hpp
+6
View File
@@ -136,6 +136,12 @@ class HeInitialization
for (size_t i = 0; i < W.n_slices; ++i)
Initialize(W.slice(i));
}
template<typename Archive>
void serialize(Archive& /* ar */, const uint32_t /* version */)
{
// Nothing to do.
}
}; // class HeInitialization
} // namespace ann
src/mlpack/methods/ann/init_rules/network_init.hpp
+13 -34
View File
@@ -14,14 +14,10 @@
#define MLPACK_METHODS_ANN_INIT_RULES_NETWORK_INIT_HPP
#include <mlpack/prereqs.hpp>
#include <mlpack/methods/ann/layer/layer.hpp>
#include "../visitor/reset_visitor.hpp"
#include "../visitor/weight_size_visitor.hpp"
#include "../visitor/weight_set_visitor.hpp"
#include "init_rules_traits.hpp"
#include <mlpack/methods/ann/layer/layer_types.hpp>
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -29,7 +25,7 @@ namespace ann /** Artificial Neural Network. */ {
* This class is used to initialize the network with the given initialization
* rule.
*/
template<typename InitializationRuleType, typename... CustomLayers>
template<typename InitializationRuleType>
class NetworkInitialization
{
public:
@@ -54,16 +50,18 @@ class NetworkInitialization
* @param parameterOffset Offset for network paramater, default 0.
*/
template <typename eT>
void Initialize(const std::vector<LayerTypes<CustomLayers...> >& network,
arma::Mat<eT>& parameter, size_t parameterOffset = 0)
void Initialize(const std::vector<Layer<arma::Mat<eT>>*>& network,
arma::Mat<eT>& parameters,
size_t parameterOffset = 0)
{
// Determine the number of parameter/weights of the given network.
if (parameter.is_empty())
// Determine the total number of parameters/weights of the given network.
if (parameters.is_empty())
{
size_t weights = 0;
for (size_t i = 0; i < network.size(); ++i)
weights += boost::apply_visitor(weightSizeVisitor, network[i]);
parameter.set_size(weights, 1);
weights += network[i]->WeightSize();
parameters.set_size(weights, 1);
}
// Initialize the network layer by layer or the complete network.
@@ -73,9 +71,8 @@ class NetworkInitialization
{
// Initialize the layer with the specified parameter/weight
// initialization rule.
const size_t weight = boost::apply_visitor(weightSizeVisitor,
network[i]);
arma::Mat<eT> tmp = arma::mat(parameter.memptr() + offset,
const size_t weight = network[i]->WeightSize();
arma::Mat<eT> tmp = arma::mat(parameters.memptr() + offset,
weight, 1, false, false);
initializeRule.Initialize(tmp, tmp.n_elem, 1);
@@ -85,19 +82,7 @@ class NetworkInitialization
}
else
{
initializeRule.Initialize(parameter, parameter.n_elem, 1);
}
// Note: We can't merge the for loop into the for loop above because
// WeightSetVisitor also sets the parameter/weights of the inner modules.
// Inner Modules are held by the parent module e.g. the concat module can
// hold various other modules.
for (size_t i = 0, offset = parameterOffset; i < network.size(); ++i)
{
offset += boost::apply_visitor(WeightSetVisitor(parameter, offset),
network[i]);
boost::apply_visitor(resetVisitor, network[i]);
initializeRule.Initialize(parameters, parameters.n_elem, 1);
}
}
@@ -105,12 +90,6 @@ class NetworkInitialization
//! Instantiated InitializationRule object for initializing the network
//! parameter.
InitializationRuleType initializeRule;
//! Locally-stored reset visitor.
ResetVisitor resetVisitor;
//! Locally-stored weight size visitor.
WeightSizeVisitor weightSizeVisitor;
}; // class NetworkInitialization
} // namespace ann
src/mlpack/methods/ann/init_rules/oivs_init.hpp
+4 -6
View File
@@ -47,15 +47,13 @@ namespace ann /** Artificial Neural Network. */ {
* w_i &=& \hat{w} \cdot \sqrt{a_i + 1}
* @f}
*
* Where f is the transfer function epsilon, k custom parameters, n the number of
* neurons in the outgoing layer and gamma a parameter that defines the random
* interval.
* Where f is the transfer function epsilon, k custom parameters, n the number
* of neurons in the outgoing layer and gamma a parameter that defines the
* random interval.
*
* @tparam ActivationFunction The activation function used for the oivs method.
*/
template<
class ActivationFunction = LogisticFunction
>
template<class ActivationFunction = LogisticFunction>
class OivsInitialization
{
public:
src/mlpack/methods/ann/init_rules/random_init.hpp
+7
View File
@@ -115,6 +115,13 @@ class RandomInitialization
Initialize(W.slice(i));
}
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */)
{
ar(CEREAL_NVP(lowerBound));
ar(CEREAL_NVP(upperBound));
}
private:
//! The number used as lower bound.
double lowerBound;
src/mlpack/methods/ann/layer/CMakeLists.txt
+6 -106
View File
@@ -3,67 +3,18 @@
set(SOURCES
add.hpp
add_impl.hpp
add_merge.hpp
add_merge_impl.hpp
adaptive_max_pooling.hpp
adaptive_max_pooling_impl.hpp
adaptive_mean_pooling.hpp
adaptive_mean_pooling_impl.hpp
alpha_dropout.hpp
alpha_dropout_impl.hpp
atrous_convolution.hpp
atrous_convolution_impl.hpp
base_layer.hpp
batch_norm.hpp
batch_norm_impl.hpp
bicubic_interpolation.hpp
bicubic_interpolation_impl.hpp
bilinear_interpolation.hpp
bilinear_interpolation_impl.hpp
channel_shuffle.hpp
channel_shuffle_impl.hpp
concat.hpp
concat_impl.hpp
concat_performance.hpp
concat_performance_impl.hpp
concatenate.hpp
concatenate_impl.hpp
constant.hpp
constant_impl.hpp
convolution.hpp
convolution_impl.hpp
dropconnect.hpp
dropconnect_impl.hpp
dropout.hpp
dropout_impl.hpp
elu.hpp
elu_impl.hpp
fast_lstm.hpp
fast_lstm_impl.hpp
flatten_t_swish.hpp
flatten_t_swish_impl.hpp
flexible_relu.hpp
flexible_relu_impl.hpp
glimpse.hpp
glimpse_impl.hpp
group_norm.hpp
group_norm_impl.hpp
gru.hpp
gru_impl.hpp
hard_tanh.hpp
hard_tanh_impl.hpp
highway.hpp
highway_impl.hpp
instance_norm.hpp
instance_norm_impl.hpp
isrlu.hpp
isrlu_impl.hpp
join.hpp
join_impl.hpp
layer.hpp
layer_norm.hpp
layer_norm_impl.hpp
layer_traits.hpp
layer_types.hpp
leaky_relu.hpp
leaky_relu_impl.hpp
@@ -71,73 +22,22 @@ set(SOURCES
linear_impl.hpp
linear_no_bias.hpp
linear_no_bias_impl.hpp
linear3d.hpp
linear3d_impl.hpp
log_softmax.hpp
log_softmax_impl.hpp
lookup.hpp
lookup_impl.hpp
lp_pooling.hpp
lp_pooling_impl.hpp
lstm.hpp
lstm_impl.hpp
max_pooling.hpp
max_pooling_impl.hpp
mean_pooling.hpp
mean_pooling_impl.hpp
minibatch_discrimination.hpp
minibatch_discrimination_impl.hpp
multihead_attention_impl.hpp
multihead_attention.hpp
multiply_constant.hpp
multiply_constant_impl.hpp
multiply_merge.hpp
multiply_merge_impl.hpp
nearest_interpolation.hpp
nearest_interpolation_impl.hpp
multi_layer.hpp
multi_layer_impl.hpp
noisylinear.hpp
noisylinear_impl.hpp
parametric_relu.hpp
parametric_relu_impl.hpp
pixel_shuffle.hpp
pixel_shuffle_impl.hpp
positional_encoding.hpp
positional_encoding_impl.hpp
recurrent.hpp
recurrent_impl.hpp
recurrent_attention.hpp
recurrent_attention_impl.hpp
reinforce_normal.hpp
reinforce_normal_impl.hpp
relu6.hpp
relu6_impl.hpp
reparametrization.hpp
reparametrization_impl.hpp
padding.hpp
radial_basis_function.hpp
radial_basis_function_impl.hpp
select.hpp
select_impl.hpp
sequential.hpp
sequential_impl.hpp
softmax_impl.hpp
softmax.hpp
spatial_dropout.hpp
spatial_dropout_impl.hpp
subview.hpp
transposed_convolution.hpp
transposed_convolution_impl.hpp
vr_class_reward.hpp
vr_class_reward_impl.hpp
c_relu.hpp
c_relu_impl.hpp
weight_norm.hpp
weight_norm_impl.hpp
hardshrink.hpp
hardshrink_impl.hpp
celu.hpp
celu_impl.hpp
softshrink.hpp
softshrink_impl.hpp
softmin.hpp
softmin_impl.hpp
serialization.hpp
)
# Add directory name to sources.
src/mlpack/methods/ann/layer/add.hpp
+51 -65
View File
@@ -13,98 +13,90 @@
#define MLPACK_METHODS_ANN_LAYER_ADD_HPP
#include <mlpack/prereqs.hpp>
#include <mlpack/methods/ann/layer/layer_traits.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
/**
* Implementation of the Add module class. The Add module applies a bias term
* to the incoming data.
* Implementation of the Add layer. The Add module applies a bias term to the
* incoming data.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class Add
template<typename MatType>
class AddType : public Layer<MatType>
{
public:
/**
* Create the Add object using the specified number of output units.
*
* @param outSize The number of output units.
* Create the AddType object. The output size of the layer will be the same
* as the input size.
*/
Add(const size_t outSize = 0);
AddType();
//! Clone the AddType object. This handles polymorphism correctly.
AddType* Clone() const { return new AddType(*this); }
// Virtual destructor.
virtual ~AddType() { }
//! Copy the given AddType layer.
AddType(const AddType& other);
//! Take ownership of the given AddType layer.
AddType(AddType&& other);
//! Copy the given AddType layer.
AddType& operator=(const AddType& other);
//! Take ownership of the given AddType layer.
AddType& operator=(AddType&& other);
/**
* Ordinary feed forward pass of a neural network, evaluating the function
* f(x) by propagating the activity forward through f.
* Forward pass: add the bias to the input.
*
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of a neural network, calculating the function
* f(x) by propagating x backwards trough f. Using the results from the feed
* forward pass.
* Backward pass: send weights backwards (the bias does not affect anything).
*
* @param * (input) The propagated input activation.
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
void Backward(const MatType& /* input */,
const MatType& gy,
MatType& g);
/**
* Calculate the gradient using the output delta and the input activation.
* Calculate the gradient using the output and the input activation.
*
* @param * (input) The propagated input.
* @param error The calculated error.
* @param gradient The calculated gradient.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient);
void Gradient(const MatType& /* input */,
const MatType& error,
MatType& gradient);
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the gradient.
OutputDataType const& Gradient() const { return gradient; }
//! Modify the gradient.
OutputDataType& Gradient() { return gradient; }
//! Get the output size.
size_t OutputSize() const { return outSize; }
//! Return the weights of the network.
const MatType& Parameters() const { return weights; }
//! Modify the weights of the network.
MatType& Parameters() { return weights; }
//! Get the size of weights.
size_t WeightSize() const { return outSize; }
//! Compute the output dimensions of the layer, based on the internal values
//! of `InputDimensions()`.
void ComputeOutputDimensions();
//! Set the weights of the layer to use the given memory.
void SetWeights(typename MatType::elem_type* weightPtr);
/**
* Serialize the layer
* Serialize the layer.
*/
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
@@ -114,18 +106,12 @@ class Add
size_t outSize;
//! Locally-stored weight object.
OutputDataType weights;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored gradient object.
OutputDataType gradient;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
MatType weights;
}; // class Add
// Standard Add layer.
typedef AddType<arma::mat> Add;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/add_impl.hpp
+81 -29
View File
@@ -19,51 +19,103 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
Add<InputDataType, OutputDataType>::Add(const size_t outSize) :
outSize(outSize)
template<typename MatType>
AddType<MatType>::AddType() : outSize(0)
{
weights.set_size(WeightSize(), 1);
// Nothing to do.
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void Add<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
template<typename MatType>
AddType<MatType>::AddType(const AddType& other) :
Layer<MatType>(other),
outSize(other.outSize)
{
output = input;
output.each_col() += weights;
// Nothing to do.
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void Add<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g)
template<typename MatType>
AddType<MatType>::AddType(AddType&& other) :
Layer<MatType>(std::move(other)),
outSize(std::move(other.outSize))
{
// Nothing to do.
}
template<typename MatType>
AddType<MatType>&
AddType<MatType>::operator=(const AddType& other)
{
if (&other != this)
{
Layer<MatType>::operator=(other);
outSize = other.outSize;
}
return *this;
}
template<typename MatType>
AddType<MatType>&
AddType<MatType>::operator=(AddType&& other)
{
if (&other != this)
{
Layer<MatType>::operator=(std::move(other));
outSize = std::move(other.outSize);
}
return *this;
}
template<typename MatType>
void AddType<MatType>::Forward(const MatType& input, MatType& output)
{
output = input + arma::repmat(arma::vectorise(weights), 1, input.n_cols);
}
template<typename MatType>
void AddType<MatType>::Backward(
const MatType& /* input */,
const MatType& gy,
MatType& g)
{
g = gy;
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void Add<InputDataType, OutputDataType>::Gradient(
const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient)
template<typename MatType>
void AddType<MatType>::Gradient(
const MatType& /* input */,
const MatType& error,
MatType& gradient)
{
gradient = error;
}
template<typename InputDataType, typename OutputDataType>
template<typename Archive>
void Add<InputDataType, OutputDataType>::serialize(
Archive& ar, const uint32_t /* version */)
template<typename MatType>
void AddType<MatType>::SetWeights(typename MatType::elem_type* weightPtr)
{
ar(CEREAL_NVP(outSize));
// Set the weights to wrap the given memory.
MakeAlias(weights, weightPtr, 1, outSize);
}
if (cereal::is_loading<Archive>())
weights.set_size(outSize, 1);
template<typename MatType>
void AddType<MatType>::ComputeOutputDimensions()
{
this->outputDimensions = this->inputDimensions;
outSize = this->outputDimensions[0];
for (size_t i = 1; i < this->outputDimensions.size(); ++i)
outSize *= this->outputDimensions[i];
}
template<typename MatType>
template<typename Archive>
void AddType<MatType>::serialize(Archive& ar, const uint32_t /* version */)
{
ar(cereal::base_class<Layer<MatType>>(this));
ar(CEREAL_NVP(outSize));
ar(CEREAL_NVP(weights));
}
} // namespace ann
src/mlpack/methods/ann/layer/add_merge_impl.hpp
-168
View File
@@ -1,168 +0,0 @@
/**
* @file methods/ann/layer/add_merge_impl.hpp
* @author Marcus Edel
*
* Definition of the AddMerge module which accumulates the output of the given
* modules.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_ADD_MERGE_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_ADD_MERGE_IMPL_HPP
// In case it hasn't yet been included.
#include "add_merge.hpp"
#include "../visitor/forward_visitor.hpp"
#include "../visitor/backward_visitor.hpp"
#include "../visitor/gradient_visitor.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
AddMerge<InputDataType, OutputDataType, CustomLayers...>::AddMerge(
const bool model, const bool run) :
model(model), run(run), ownsLayers(!model)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
AddMerge<InputDataType, OutputDataType, CustomLayers...>::AddMerge(
const bool model, const bool run, const bool ownsLayers) :
model(model), run(run), ownsLayers(ownsLayers)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
AddMerge<InputDataType, OutputDataType, CustomLayers...>::~AddMerge()
{
if (!model && ownsLayers)
{
std::for_each(network.begin(), network.end(),
boost::apply_visitor(deleteVisitor));
}
}
template <typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename InputType, typename OutputType>
void AddMerge<InputDataType, OutputDataType, CustomLayers...>::Forward(
const InputType& input, OutputType& output)
{
if (run)
{
for (size_t i = 0; i < network.size(); ++i)
{
boost::apply_visitor(ForwardVisitor(input,
boost::apply_visitor(outputParameterVisitor, network[i])),
network[i]);
}
}
output = boost::apply_visitor(outputParameterVisitor, network.front());
for (size_t i = 1; i < network.size(); ++i)
{
output += boost::apply_visitor(outputParameterVisitor, network[i]);
}
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void AddMerge<InputDataType, OutputDataType, CustomLayers...>::Backward(
const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g)
{
if (run)
{
for (size_t i = 0; i < network.size(); ++i)
{
boost::apply_visitor(BackwardVisitor(boost::apply_visitor(
outputParameterVisitor, network[i]), gy,
boost::apply_visitor(deltaVisitor, network[i])), network[i]);
}
g = boost::apply_visitor(deltaVisitor, network[0]);
for (size_t i = 1; i < network.size(); ++i)
{
g += boost::apply_visitor(deltaVisitor, network[i]);
}
}
else
g = gy;
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void AddMerge<InputDataType, OutputDataType, CustomLayers...>::Backward(
const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g,
const size_t index)
{
boost::apply_visitor(BackwardVisitor(boost::apply_visitor(
outputParameterVisitor, network[index]), gy,
boost::apply_visitor(deltaVisitor, network[index])), network[index]);
g = boost::apply_visitor(deltaVisitor, network[index]);
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void AddMerge<InputDataType, OutputDataType, CustomLayers...>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& /* gradient */ )
{
if (run)
{
for (size_t i = 0; i < network.size(); ++i)
{
boost::apply_visitor(GradientVisitor(input, error), network[i]);
}
}
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void AddMerge<InputDataType, OutputDataType, CustomLayers...>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& /* gradient */,
const size_t index)
{
boost::apply_visitor(GradientVisitor(input, error), network[index]);
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename Archive>
void AddMerge<InputDataType, OutputDataType, CustomLayers...>::serialize(
Archive& ar, const uint32_t /* version */)
{
// Be sure to clear other layers before loading.
if (cereal::is_loading<Archive>())
network.clear();
ar(CEREAL_VECTOR_VARIANT_POINTER(network));
ar(CEREAL_NVP(model));
ar(CEREAL_NVP(run));
ar(CEREAL_NVP(ownsLayers));
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/alpha_dropout.hpp
+34 -45
View File
@@ -17,6 +17,7 @@
#define MLPACK_METHODS_ANN_LAYER_ALPHA_DROPOUT_HPP
#include <mlpack/prereqs.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -40,14 +41,11 @@ namespace ann /** Artificial Neural Network. */ {
* }
* @endcode
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template <typename InputDataType = arma::mat,
typename OutputDataType = arma::mat>
class AlphaDropout
template<typename MatType = arma::mat>
class AlphaDropoutType : public Layer<MatType>
{
public:
/**
@@ -56,17 +54,33 @@ class AlphaDropout
* @param ratio The probability of setting a value to alphaDash.
* @param alphaDash The dropout scaling parameter.
*/
AlphaDropout(const double ratio = 0.5,
const double alphaDash = -alpha * lambda);
AlphaDropoutType(const double ratio = 0.5,
const double alphaDash = -alpha * lambda);
/**
* Ordinary feed forward pass of the alpha_dropout layer.
* Clone the AlphaDropoutType object. This handles polymorphism correctly.
*/
AlphaDropoutType* Clone() const { return new AlphaDropoutType(*this); }
// Virtual destructor.
virtual ~AlphaDropoutType() { }
//! Copy the given AlphaDropoutType layer.
AlphaDropoutType(const AlphaDropoutType& other);
//! Take ownership of the given AlphaDropoutType layer.
AlphaDropoutType(AlphaDropoutType&& other);
//! Copy the given AlphaDropoutType layer.
AlphaDropoutType& operator=(const AlphaDropoutType& other);
//! Take ownership of the given AlphaDropoutType layer.
AlphaDropoutType& operator=(AlphaDropoutType&& other);
/**
* Ordinary feed forward pass of the AlphaDropout layer.
*
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of the alpha_dropout layer.
@@ -75,25 +89,7 @@ class AlphaDropout
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the detla.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! The value of the deterministic parameter.
bool Deterministic() const { return deterministic; }
//! Modify the value of the deterministic parameter.
bool& Deterministic() { return deterministic; }
void Backward(const MatType& /* input */, const MatType& gy, MatType& g);
//! The probability of setting a value to alphaDash.
double Ratio() const { return ratio; }
@@ -105,10 +101,10 @@ class AlphaDropout
double B() const { return b; }
//! Value of alphaDash.
double AlphaDash() const {return alphaDash; }
double AlphaDash() const { return alphaDash; }
//! Get the mask.
OutputDataType const& Mask() const {return mask;}
const MatType& Mask() const { return mask; }
//! Modify the probability of setting a value to alphaDash. As
//! 'a' and 'b' depend on 'ratio', modify them as well.
@@ -126,14 +122,8 @@ class AlphaDropout
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
//! Locally-stored mast object.
OutputDataType mask;
//! Locally-stored mask object.
MatType mask;
//! The probability of setting a value to aplhaDash.
double ratio;
@@ -141,9 +131,6 @@ class AlphaDropout
//! The low variance value of SELU activation function.
double alphaDash;
//! If true dropout and scaling is disabled, see notes above.
bool deterministic;
//! Value of alpha for normalized inputs (taken from SELU).
static constexpr double alpha = 1.6732632423543772848170429916717;
@@ -155,7 +142,9 @@ class AlphaDropout
//! Value to be added to a*x for affine transformation.
double b;
}; // class AlphaDropout
}; // class AlphaDropoutType
typedef AlphaDropoutType<arma::mat> AlphaDropout;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/alpha_dropout_impl.hpp
+78 -18
View File
@@ -22,25 +22,79 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
AlphaDropout<InputDataType, OutputDataType>::AlphaDropout(
template<typename MatType>
AlphaDropoutType<MatType>::AlphaDropoutType(
const double ratio,
const double alphaDash) :
ratio(ratio),
alphaDash(alphaDash),
deterministic(false)
alphaDash(alphaDash)
{
Ratio(ratio);
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void AlphaDropout<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
template<typename MatType>
AlphaDropoutType<MatType>::AlphaDropoutType(const AlphaDropoutType& other) :
Layer<MatType>(other),
mask(other.mask),
ratio(other.ratio),
alphaDash(other.alphaDash),
a(other.a),
b(other.b)
{
// The dropout mask will not be multiplied in the deterministic mode
// (during testing).
if (deterministic)
// Nothing to do.
}
template<typename MatType>
AlphaDropoutType<MatType>::AlphaDropoutType(AlphaDropoutType&& other) :
Layer<MatType>(std::move(other)),
mask(std::move(other.mask)),
ratio(std::move(other.ratio)),
alphaDash(std::move(other.alphaDash)),
a(std::move(other.a)),
b(std::move(other.b))
{
// Nothing to do.
}
template<typename MatType>
AlphaDropoutType<MatType>&
AlphaDropoutType<MatType>::operator=(const AlphaDropoutType& other)
{
if (&other != this)
{
Layer<MatType>::operator=(other);
mask = other.mask;
ratio = other.ratio;
alphaDash = other.alphaDash;
a = other.a;
b = other.b;
}
return *this;
}
template<typename MatType>
AlphaDropoutType<MatType>&
AlphaDropoutType<MatType>::operator=(AlphaDropoutType&& other)
{
if (&other != this)
{
Layer<MatType>::operator=(std::move(other));
mask = std::move(other.mask);
ratio = std::move(other.ratio);
alphaDash = std::move(other.alphaDash);
a = std::move(other.a);
b = std::move(other.b);
}
return *this;
}
template<typename MatType>
void AlphaDropoutType<MatType>::Forward(const MatType& input, MatType& output)
{
// The dropout mask will not be multiplied during testing.
if (!this->training)
{
output = input;
}
@@ -49,29 +103,35 @@ void AlphaDropout<InputDataType, OutputDataType>::Forward(
// Set values to alphaDash with probability ratio. Then apply affine
// transformation so as to keep mean and variance of outputs to their
// original values.
mask = arma::randu< arma::Mat<eT> >(input.n_rows, input.n_cols);
mask = arma::randu<MatType>(input.n_rows, input.n_cols);
mask.transform( [&](double val) { return (val > ratio); } );
output = (input % mask + alphaDash * (1 - mask)) * a + b;
}
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void AlphaDropout<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& /* input */, const arma::Mat<eT>& gy, arma::Mat<eT>& g)
template<typename MatType>
void AlphaDropoutType<MatType>::Backward(
const MatType& /* input */, const MatType& gy, MatType& g)
{
g = gy % mask * a;
}
template<typename InputDataType, typename OutputDataType>
template<typename MatType>
template<typename Archive>
void AlphaDropout<InputDataType, OutputDataType>::serialize(
void AlphaDropoutType<MatType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(cereal::base_class<Layer<MatType>>(this));
ar(CEREAL_NVP(ratio));
ar(CEREAL_NVP(alphaDash));
ar(CEREAL_NVP(a));
ar(CEREAL_NVP(b));
// No need to serialize the mask, since it will be recomputed on the next
// forward pass. But we should clear it if we are loading.
if (Archive::is_loading::value)
mask.clear();
}
} // namespace ann
src/mlpack/methods/ann/layer/base_layer.hpp
+96 -171
View File
@@ -30,6 +30,7 @@
#include <mlpack/methods/ann/activation_functions/hard_swish_function.hpp>
#include <mlpack/methods/ann/activation_functions/tanh_exponential_function.hpp>
#include <mlpack/methods/ann/activation_functions/silu_function.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -40,102 +41,83 @@ namespace ann /** Artificial Neural Network. */ {
*
* A few convenience typedefs are given:
*
* - SigmoidLayer
* - IdentityLayer
* - ReLULayer
* - TanHLayer
* - SoftplusLayer
* - HardSigmoidLayer
* - SwishLayer
* - MishLayer
* - LiSHTLayer
* - GELULayer
* - ELiSHLayer
* - ElliotLayer
* - GaussianLayer
* - HardSwishLayer
* - TanhExpLayer
* - SILULayer
* - Sigmoid
* - ReLU
* - TanH
* - Softplus
* - HardSigmoid
* - Swish
* - Mish
* - LiSHT
* - GELU
* - ELiSH
* - Elliot
* - Gaussian
* - HardSwish
* - TanhExp
* - SILU
*
* @tparam ActivationFunction Activation function used for the embedding layer.
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
*/
template <
class ActivationFunction = LogisticFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
typename MatType = arma::mat
>
class BaseLayer
class BaseLayer : public Layer<MatType>
{
public:
/**
* Create the BaseLayer object.
*/
BaseLayer()
BaseLayer() : Layer<MatType>()
{
// Nothing to do here.
}
// Virtual destructor.
virtual ~BaseLayer() { }
// No copy constructor or operators needed here, since the class has no
// members.
//! Clone the BaseLayer object. This handles polymorphism correctly.
BaseLayer* Clone() const { return new BaseLayer(*this); }
/**
* Ordinary feed forward pass of a neural network, evaluating the function
* f(x) by propagating the activity forward through f.
* Forward pass: apply the activation to the inputs.
*
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename InputType, typename OutputType>
void Forward(const InputType& input, OutputType& output)
void Forward(const MatType& input, MatType& output)
{
ActivationFunction::Fn(input, output);
}
/**
* Ordinary feed backward pass of a neural network, calculating the function
* f(x) by propagating x backwards trough f. Using the results from the feed
* forward pass.
* Backward pass: compute the function f(x) by propagating x backwards through
* f, using the results from the forward pass.
*
* @param input The propagated input activation.
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& input,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g)
void Backward(const MatType& input, const MatType& gy, MatType& g)
{
arma::Mat<eT> derivative;
MatType derivative;
ActivationFunction::Deriv(input, derivative);
g = gy % derivative;
}
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
/**
* Serialize the layer.
*/
template<typename Archive>
void serialize(Archive& /* ar */, const uint32_t /* version */)
void serialize(Archive& ar, const uint32_t /* version */)
{
/* Nothing to do here */
ar(cereal::base_class<Layer<MatType>>(this));
// Nothing to serialize.
}
private:
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
}; // class BaseLayer
// Convenience typedefs.
@@ -143,179 +125,122 @@ class BaseLayer
/**
* Standard Sigmoid-Layer using the logistic activation function.
*/
template <
class ActivationFunction = LogisticFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using SigmoidLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<LogisticFunction, arma::mat> Sigmoid;
/**
* Standard Identity-Layer using the identity activation function.
*/
template <
class ActivationFunction = IdentityFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using IdentityLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
template<typename MatType = arma::mat>
using SigmoidType = BaseLayer<LogisticFunction, MatType>;
/**
* Standard rectified linear unit non-linearity layer.
*/
template <
class ActivationFunction = RectifierFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using ReLULayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<RectifierFunction, arma::mat> ReLU;
template<typename MatType = arma::mat>
using ReLUType = BaseLayer<RectifierFunction, MatType>;
/**
* Standard hyperbolic tangent layer.
*/
template <
class ActivationFunction = TanhFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using TanHLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<TanhFunction, arma::mat> TanH;
template<typename MatType = arma::mat>
using TanHType = BaseLayer<TanhFunction, MatType>;
/**
* Standard Softplus-Layer using the Softplus activation function.
*/
template <
class ActivationFunction = SoftplusFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using SoftPlusLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<SoftplusFunction, arma::mat> SoftPlus;
template<typename MatType = arma::mat>
using SoftPlusType = BaseLayer<SoftplusFunction, MatType>;
/**
* Standard HardSigmoid-Layer using the HardSigmoid activation function.
*/
template <
class ActivationFunction = HardSigmoidFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using HardSigmoidLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<HardSigmoidFunction, arma::mat> HardSigmoid;
template<typename MatType = arma::mat>
using HardSigmoidType = BaseLayer<HardSigmoidFunction, MatType>;
/**
* Standard Swish-Layer using the Swish activation function.
*/
template <
class ActivationFunction = SwishFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using SwishFunctionLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<SwishFunction, arma::mat> Swish;
template<typename MatType = arma::mat>
using SwishType = BaseLayer<SwishFunction, MatType>;
/**
* Standard Mish-Layer using the Mish activation function.
*/
template <
class ActivationFunction = MishFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using MishFunctionLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<MishFunction, arma::mat> Mish;
template<typename MatType = arma::mat>
using MishType = BaseLayer<MishFunction, MatType>;
/**
* Standard LiSHT-Layer using the LiSHT activation function.
*/
template <
class ActivationFunction = LiSHTFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using LiSHTFunctionLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<LiSHTFunction, arma::mat> LiSHT;
template<typename MatType = arma::mat>
using LiSHTType = BaseLayer<LiSHTFunction, MatType>;
/**
* Standard GELU-Layer using the GELU activation function.
*/
template <
class ActivationFunction = GELUFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using GELUFunctionLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<GELUFunction, arma::mat> GELU;
template<typename MatType = arma::mat>
using GELUType = BaseLayer<GELUFunction, MatType>;
/**
* Standard Elliot-Layer using the Elliot activation function.
*/
template <
class ActivationFunction = ElliotFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using ElliotFunctionLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<ElliotFunction, arma::mat> Elliot;
template<typename MatType = arma::mat>
using ElliotType = BaseLayer<ElliotFunction, MatType>;
/**
* Standard ELiSH-Layer using the ELiSH activation function.
*/
template <
class ActivationFunction = ElishFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using ElishFunctionLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<ElishFunction, arma::mat> Elish;
template<typename MatType = arma::mat>
using ElishType = BaseLayer<ElishFunction, MatType>;
/**
* Standard Gaussian-Layer using the Gaussian activation function.
*/
template <
class ActivationFunction = GaussianFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using GaussianFunctionLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<GaussianFunction, arma::mat> Gaussian;
template<typename MatType = arma::mat>
using GaussianType = BaseLayer<GaussianFunction, MatType>;
/**
* Standard HardSwish-Layer using the HardSwish activation function.
*/
template <
class ActivationFunction = HardSwishFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using HardSwishFunctionLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<HardSwishFunction, arma::mat> HardSwish;
template <typename MatType = arma::mat>
using HardSwishType = BaseLayer<HardSwishFunction, MatType>;
/**
* Standard TanhExp-Layer using the TanhExp activation function.
*/
template <
class ActivationFunction = TanhExpFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using TanhExpFunctionLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType>;
typedef BaseLayer<TanhExpFunction, arma::mat> TanhExp;
template<typename MatType = arma::mat>
using TanhExpType = BaseLayer<TanhExpFunction, MatType>;
/**
* Standard SILU-Layer using the SILU activation function.
*/
template <
class ActivationFunction = SILUFunction,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
using SILUFunctionLayer = BaseLayer<
ActivationFunction, InputDataType, OutputDataType
>;
typedef BaseLayer<SILUFunction, arma::mat> SILU;
template<typename MatType = arma::mat>
using SILUType = BaseLayer<SILUFunction, MatType>;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/concat.hpp
-263
View File
@@ -1,263 +0,0 @@
/**
* @file methods/ann/layer/concat.hpp
* @author Marcus Edel
* @author Mehul Kumar Nirala
*
* Definition of the Concat class, which acts as a concatenation container.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_CONCAT_HPP
#define MLPACK_METHODS_ANN_LAYER_CONCAT_HPP
#include <mlpack/prereqs.hpp>
#include "../visitor/delete_visitor.hpp"
#include "../visitor/delta_visitor.hpp"
#include "../visitor/output_parameter_visitor.hpp"
#include "layer_types.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
/**
* Implementation of the Concat class. The Concat class works as a
* feed-forward fully connected network container which plugs various layers
* together.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam CustomLayers Additional custom layers if required.
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat,
typename... CustomLayers
>
class Concat
{
public:
/**
* Create the Concat object using the specified parameters.
*
* @param model Expose all network modules.
* @param run Call the Forward/Backward method before the output is merged.
*/
Concat(const bool model = false,
const bool run = true);
/**
* Create the Concat object using the specified parameters.
*
* @param inputSize A vector denoting input size of each layer added.
* @param axis Concat axis.
* @param model Expose all network modules.
* @param run Call the Forward/Backward method before the output is merged.
*/
Concat(arma::Row<size_t>& inputSize,
const size_t axis,
const bool model = false,
const bool run = true);
/**
* Destroy the layers held by the model.
*/
~Concat();
/**
* Ordinary feed forward pass of a neural network, evaluating the function
* f(x) by propagating the activity forward through f.
*
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
/**
* Ordinary feed backward pass of a neural network, using 3rd-order tensors as
* input, calculating the function f(x) by propagating x backwards through f.
* Using the results from the feed forward pass.
*
* @param * (input) The propagated input activation.
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
/**
* This is the overload of Backward() that runs only a specific layer with
* the given input.
*
* @param * (input) The propagated input activation.
* @param gy The backpropagated error.
* @param g The calculated gradient.
* @param index The index of the layer to run.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g,
const size_t index);
/*
* Calculate the gradient using the output delta and the input activation.
*
* @param input The input parameter used for calculating the gradient.
* @param error The calculated error.
* @param gradient The calculated gradient.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& error,
arma::Mat<eT>& /* gradient */);
/*
* This is the overload of Gradient() that runs a specific layer with the
* given input.
*
* @param input The input parameter used for calculating the gradient.
* @param error The calculated error.
* @param gradient The calculated gradient.
* @param The index of the layer to run.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient,
const size_t index);
/*
* Add a new module to the model.
*
* @param args The layer parameter.
*/
template <class LayerType, class... Args>
void Add(Args... args) { network.push_back(new LayerType(args...)); }
/*
* Add a new module to the model.
*
* @param layer The Layer to be added to the model.
*/
void Add(LayerTypes<CustomLayers...> layer) { network.push_back(layer); }
//! Return the model modules.
std::vector<LayerTypes<CustomLayers...> >& Model()
{
if (model)
{
return network;
}
return empty;
}
//! Return the initial point for the optimization.
const arma::mat& Parameters() const { return weights; }
//! Modify the initial point for the optimization.
arma::mat& Parameters() { return weights; }
//! Get the value of run parameter.
bool Run() const { return run; }
//! Modify the value of run parameter.
bool& Run() { return run; }
arma::mat const& InputParameter() const { return inputParameter; }
//! Modify the input parameter.
arma::mat& InputParameter() { return inputParameter; }
//! Get the output parameter.
arma::mat const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
arma::mat& OutputParameter() { return outputParameter; }
//! Get the delta.e
arma::mat const& Delta() const { return delta; }
//! Modify the delta.
arma::mat& Delta() { return delta; }
//! Get the gradient.
arma::mat const& Gradient() const { return gradient; }
//! Modify the gradient.
arma::mat& Gradient() { return gradient; }
//! Get the axis of concatenation.
size_t const& ConcatAxis() const { return axis; }
//! Get the size of the weight matrix.
size_t WeightSize() const { return 0; }
/**
* Serialize the layer
*/
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Parameter which indicates the input size of modules.
arma::Row<size_t> inputSize;
//! Parameter which indicates the axis of concatenation.
size_t axis;
//! Parameter which indicates whether to use the axis of concatenation.
bool useAxis;
//! Parameter which indicates if the modules should be exposed.
bool model;
//! Parameter which indicates if the Forward/Backward method should be called
//! before merging the output.
bool run;
//! Parameter to store channels.
size_t channels;
//! Locally-stored network modules.
std::vector<LayerTypes<CustomLayers...> > network;
//! Locally-stored model weights.
OutputDataType weights;
//! Locally-stored delta visitor.
DeltaVisitor deltaVisitor;
//! Locally-stored output parameter visitor.
OutputParameterVisitor outputParameterVisitor;
//! Locally-stored delete visitor.
DeleteVisitor deleteVisitor;
//! Locally-stored empty list of modules.
std::vector<LayerTypes<CustomLayers...> > empty;
//! Locally-stored delta object.
arma::mat delta;
//! Locally-stored input parameter object.
arma::mat inputParameter;
//! Locally-stored output parameter object.
arma::mat outputParameter;
//! Locally-stored gradient object.
arma::mat gradient;
}; // class Concat
} // namespace ann
} // namespace mlpack
// Include implementation.
#include "concat_impl.hpp"
#endif
src/mlpack/methods/ann/layer/concat_impl.hpp
-293
View File
@@ -1,293 +0,0 @@
/**
* @file methods/ann/layer/concat_impl.hpp
* @author Marcus Edel
* @author Mehul Kumar Nirala
*
* Implementation of the Concat class, which acts as a concatenation contain.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_CONCAT_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_CONCAT_IMPL_HPP
// In case it hasn't yet been included.
#include "concat.hpp"
#include "../visitor/forward_visitor.hpp"
#include "../visitor/backward_visitor.hpp"
#include "../visitor/gradient_visitor.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
Concat<InputDataType, OutputDataType, CustomLayers...>::Concat(
const bool model, const bool run) :
axis(0),
useAxis(false),
model(model),
run(run),
channels(1)
{
weights.set_size(0, 0);
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
Concat<InputDataType, OutputDataType, CustomLayers...>::Concat(
arma::Row<size_t>& inputSize,
const size_t axis,
const bool model,
const bool run) :
inputSize(inputSize),
axis(axis),
useAxis(true),
model(model),
run(run)
{
weights.set_size(0, 0);
// Parameters to help calculate the number of channels.
size_t oldColSize = 1, newColSize = 1;
// Axis is specified and useAxis is true.
if (useAxis)
{
// Axis is specified without input dimension.
// Throw an error.
if (inputSize.n_elem > 0)
{
// Calculate rowSize, newColSize based on the axis
// of concatenation. Finally concat along cols and
// reshape to original format i.e. (input, batch_size).
size_t i = std::min(axis + 1, (size_t) inputSize.n_elem);
for (; i < inputSize.n_elem; ++i)
{
newColSize *= inputSize[i];
}
}
else
{
throw std::logic_error("Input dimensions not specified.");
}
}
else
{
channels = 1;
}
if (newColSize <= 0)
{
throw std::logic_error("Col size is zero.");
}
channels = newColSize / oldColSize;
inputSize.clear();
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
Concat<InputDataType, OutputDataType, CustomLayers...>::~Concat()
{
if (!model)
{
// Clear memory.
std::for_each(network.begin(), network.end(),
boost::apply_visitor(deleteVisitor));
}
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void Concat<InputDataType, OutputDataType, CustomLayers...>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
{
if (run)
{
for (size_t i = 0; i < network.size(); ++i)
{
boost::apply_visitor(ForwardVisitor(input,
boost::apply_visitor(outputParameterVisitor, network[i])),
network[i]);
}
}
output = boost::apply_visitor(outputParameterVisitor, network.front());
// Reshape output to incorporate the channels.
output.reshape(output.n_rows / channels, output.n_cols * channels);
for (size_t i = 1; i < network.size(); ++i)
{
arma::Mat<eT> out = boost::apply_visitor(outputParameterVisitor,
network[i]);
out.reshape(out.n_rows / channels, out.n_cols * channels);
// Vertically concatentate output from each layer.
output = arma::join_cols(output, out);
}
// Reshape output to its original shape.
output.reshape(output.n_rows * channels, output.n_cols / channels);
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void Concat<InputDataType, OutputDataType, CustomLayers...>::Backward(
const arma::Mat<eT>& /* input */, const arma::Mat<eT>& gy, arma::Mat<eT>& g)
{
size_t rowCount = 0;
if (run)
{
arma::Mat<eT> delta;
arma::Mat<eT> gyTmp(((arma::Mat<eT>&) gy).memptr(), gy.n_rows / channels,
gy.n_cols * channels, false, false);
for (size_t i = 0; i < network.size(); ++i)
{
// Use rows from the error corresponding to the output from each layer.
size_t rows = boost::apply_visitor(
outputParameterVisitor, network[i]).n_rows;
// Extract from gy the parameters for the i-th network.
delta = gyTmp.rows(rowCount / channels, (rowCount + rows) / channels - 1);
delta.reshape(delta.n_rows * channels, delta.n_cols / channels);
boost::apply_visitor(BackwardVisitor(
boost::apply_visitor(outputParameterVisitor,
network[i]), delta,
boost::apply_visitor(deltaVisitor, network[i])), network[i]);
rowCount += rows;
}
g = boost::apply_visitor(deltaVisitor, network[0]);
for (size_t i = 1; i < network.size(); ++i)
{
g += boost::apply_visitor(deltaVisitor, network[i]);
}
}
else
{
g = gy;
}
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void Concat<InputDataType, OutputDataType, CustomLayers...>::Backward(
const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g,
const size_t index)
{
size_t rowCount = 0, rows = 0;
for (size_t i = 0; i < index; ++i)
{
rowCount += boost::apply_visitor(
outputParameterVisitor, network[i]).n_rows;
}
rows = boost::apply_visitor(outputParameterVisitor, network[index]).n_rows;
// Reshape gy to extract the i-th layer gy.
arma::Mat<eT> gyTmp(((arma::Mat<eT>&) gy).memptr(), gy.n_rows / channels,
gy.n_cols * channels, false, false);
arma::Mat<eT> delta = gyTmp.rows(rowCount / channels, (rowCount + rows) /
channels - 1);
delta.reshape(delta.n_rows * channels, delta.n_cols / channels);
boost::apply_visitor(BackwardVisitor(boost::apply_visitor(
outputParameterVisitor, network[index]), delta,
boost::apply_visitor(deltaVisitor, network[index])), network[index]);
g = boost::apply_visitor(deltaVisitor, network[index]);
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void Concat<InputDataType, OutputDataType, CustomLayers...>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& /* gradient */)
{
if (run)
{
size_t rowCount = 0;
// Reshape error to extract the i-th layer error.
arma::Mat<eT> errorTmp(((arma::Mat<eT>&) error).memptr(),
error.n_rows / channels, error.n_cols * channels, false, false);
for (size_t i = 0; i < network.size(); ++i)
{
size_t rows = boost::apply_visitor(
outputParameterVisitor, network[i]).n_rows;
// Extract from error the parameters for the i-th network.
arma::Mat<eT> err = errorTmp.rows(rowCount / channels, (rowCount + rows) /
channels - 1);
err.reshape(err.n_rows * channels, err.n_cols / channels);
boost::apply_visitor(GradientVisitor(input, err), network[i]);
rowCount += rows;
}
}
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void Concat<InputDataType, OutputDataType, CustomLayers...>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& /* gradient */,
const size_t index)
{
size_t rowCount = 0;
for (size_t i = 0; i < index; ++i)
{
rowCount += boost::apply_visitor(outputParameterVisitor,
network[i]).n_rows;
}
size_t rows = boost::apply_visitor(
outputParameterVisitor, network[index]).n_rows;
arma::Mat<eT> errorTmp(((arma::Mat<eT>&) error).memptr(),
error.n_rows / channels, error.n_cols * channels, false, false);
arma::Mat<eT> err = errorTmp.rows(rowCount / channels, (rowCount + rows) /
channels - 1);
err.reshape(err.n_rows * channels, err.n_cols / channels);
boost::apply_visitor(GradientVisitor(input, err), network[index]);
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename Archive>
void Concat<InputDataType, OutputDataType, CustomLayers...>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(CEREAL_NVP(model));
ar(CEREAL_NVP(run));
// Do we have to load or save a model?
if (model)
{
// Clear memory first, if needed.
if (cereal::is_loading<Archive>())
{
std::for_each(network.begin(), network.end(),
boost::apply_visitor(deleteVisitor));
}
ar(CEREAL_VECTOR_VARIANT_POINTER(network));
}
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/concatenate.hpp
+39 -63
View File
@@ -14,7 +14,7 @@
#define MLPACK_METHODS_ANN_LAYER_CONCATENATE_HPP
#include <mlpack/prereqs.hpp>
#include <mlpack/methods/ann/layer/layer_traits.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -22,36 +22,39 @@ namespace ann /** Artificial Neural Network. */ {
/**
* Implementation of the Concatenate module class. The Concatenate module
* concatenates a constant given matrix to the incoming data.
* Note: Users need to use the Concat() function to provide the concat matrix.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* The Concat() function provides the concat matrix, or it can be passed to
* the constructor.
*
* After this layer is applied, the shape of the data will be a vector.
*
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class Concatenate
template<typename MatType = arma::mat>
class ConcatenateType : public Layer<MatType>
{
public:
/**
* Create the Concatenate object using the specified number of output units.
* Create the ConcatenateType object using the given constant matrix as the
* data to be concatenated to the output of the forward pass.
*/
Concatenate();
ConcatenateType(const MatType& concat = MatType());
//! Copy constructor.
Concatenate(const Concatenate& layer);
//! Clone the ConcatenateType object. This handles polymorphism correctly.
ConcatenateType* Clone() const { return new ConcatenateType(*this); }
//! Move constructor.
Concatenate(Concatenate&& layer);
// Virtual destructor.
virtual ~ConcatenateType() { }
//! Operator= copy constructor.
Concatenate& operator=(const Concatenate& layer);
//! Operator= move constructor.
Concatenate& operator=(Concatenate&& layer);
//! Copy the given ConcatenateType layer.
ConcatenateType(const ConcatenateType& other);
//! Take ownership of the given ConcatenateType layer.
ConcatenateType(ConcatenateType&& other);
//! Copy the given ConcatenateType layer.
ConcatenateType& operator=(const ConcatenateType& other);
//! Take ownership of the given ConcatenateType layer.
ConcatenateType& operator=(ConcatenateType&& other);
/**
* Ordinary feed forward pass of a neural network, evaluating the function
@@ -60,8 +63,7 @@ class Concatenate
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of a neural network, calculating the function
@@ -72,57 +74,31 @@ class Concatenate
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
void Backward(const MatType& /* input */, const MatType& gy, MatType& g);
//! Get the concat matrix.
OutputDataType const& Concat() const { return concat; }
MatType const& Concat() const { return concat; }
//! Modify the concat.
OutputDataType& Concat() { return concat; }
MatType& Concat() { return concat; }
//! Compute the output dimensions of the layer based on `InputDimensions()`.
void ComputeOutputDimensions();
/**
* Serialize the layer
* Serialize the layer.
*/
template<typename Archive>
void serialize(Archive& /* ar */, const uint32_t /* version */)
{
// Nothing to do here.
}
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Locally-stored number of input rows.
size_t inRows;
//! Matrix to be concatenated to input.
MatType concat;
//! Locally-stored weight object.
OutputDataType weights;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
//! Locally-stored matrix to be concatenated to input.
OutputDataType concat;
}; // class Concatenate
// Standard Concatenate layer.
typedef ConcatenateType<arma::mat> Concatenate;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/concatenate_impl.hpp
+69 -54
View File
@@ -19,91 +19,106 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
Concatenate<InputDataType, OutputDataType>::Concatenate() :
inRows(0)
template<typename MatType>
ConcatenateType<MatType>::
ConcatenateType(const MatType& concat) :
concat(concat)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
Concatenate<InputDataType, OutputDataType>::Concatenate(
const Concatenate& layer) :
inRows(layer.inRows),
weights(layer.weights),
delta(layer.delta),
concat(layer.concat)
template<typename MatType>
ConcatenateType<MatType>::
ConcatenateType(const ConcatenateType& other) :
Layer<MatType>(other),
concat(other.concat)
{
// Nothing to to here.
// Nothing to do.
}
template<typename InputDataType, typename OutputDataType>
Concatenate<InputDataType, OutputDataType>::Concatenate(Concatenate&& layer) :
inRows(layer.inRows),
weights(std::move(layer.weights)),
delta(std::move(layer.delta)),
concat(std::move(layer.concat))
template<typename MatType>
ConcatenateType<MatType>::
ConcatenateType(ConcatenateType&& other) :
Layer<MatType>(std::move(other)),
concat(other.concat)
{
// Nothing to do here.
// Nothing to do.
}
template<typename InputDataType, typename OutputDataType>
Concatenate<InputDataType, OutputDataType>&
Concatenate<InputDataType, OutputDataType>::
operator=(const Concatenate& layer)
template<typename MatType>
ConcatenateType<MatType>&
ConcatenateType<MatType>::operator=(const ConcatenateType& other)
{
if (this != &layer)
if (&other != this)
{
inRows = layer.inRows;
weights = layer.weights;
delta = layer.delta;
concat = layer.concat;
Layer<MatType>::operator=(other);
concat = other.concat;
}
return *this;
}
template<typename InputDataType, typename OutputDataType>
Concatenate<InputDataType, OutputDataType>&
Concatenate<InputDataType, OutputDataType>::
operator=(Concatenate&& layer)
template<typename MatType>
ConcatenateType<MatType>&
ConcatenateType<MatType>::operator=(ConcatenateType&& other)
{
if (this != &layer)
if (&other != this)
{
inRows = layer.inRows;
weights = std::move(layer.weights);
delta = std::move(layer.delta);
concat = std::move(layer.concat);
Layer<MatType>::operator=(std::move(other));
concat = std::move(other.concat);
}
return *this;
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void Concatenate<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
template<typename MatType>
void ConcatenateType<MatType>::Forward(const MatType& input, MatType& output)
{
if (concat.is_empty())
Log::Warn << "The concat matrix has not been provided." << std::endl;
if (input.n_cols != concat.n_cols)
{
Log::Fatal << "The number of columns of the concat matrix should be equal "
<< "to the number of columns of input matrix." << std::endl;
Log::Warn << "Concatenate::Forward(): the concat matrix is empty or was "
<< "not provided." << std::endl;
}
inRows = input.n_rows;
output = arma::join_cols(input, concat);
output.submat(0, 0, input.n_rows - 1, input.n_cols - 1) = input;
output.submat(input.n_rows, 0, output.n_rows - 1, input.n_cols - 1) =
arma::repmat(arma::vectorise(concat), 1, input.n_cols);
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void Concatenate<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g)
template<typename MatType>
void ConcatenateType<MatType>::Backward(
const MatType& /* input */,
const MatType& gy,
MatType& g)
{
g = gy.submat(0, 0, inRows - 1, concat.n_cols - 1);
// Pass back the non-concatenated part.
g = gy.submat(0, 0, gy.n_rows - 1 - concat.n_elem, gy.n_cols - 1);
}
template<typename MatType>
void ConcatenateType<MatType>::ComputeOutputDimensions()
{
// This flattens the input.
size_t inSize = this->inputDimensions[0];
for (size_t i = 1; i < this->inputDimensions.size(); ++i)
inSize *= this->inputDimensions[i];
this->outputDimensions = std::vector<size_t>(this->inputDimensions.size(),
1);
this->outputDimensions[0] = inSize + concat.n_elem;
}
/**
* Serialize the layer.
*/
template<typename MatType>
template<typename Archive>
void ConcatenateType<MatType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(cereal::base_class<Layer<MatType>>(this));
ar(CEREAL_NVP(concat));
}
} // namespace ann
src/mlpack/methods/ann/layer/constant_impl.hpp
-65
View File
@@ -1,65 +0,0 @@
/**
* @file methods/ann/layer/constant_impl.hpp
* @author Marcus Edel
*
* Implementation of the Constant class, which outputs a constant value given
* any input.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_CONSTANT_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_CONSTANT_IMPL_HPP
// In case it hasn't yet been included.
#include "constant.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
Constant<InputDataType, OutputDataType>::Constant(
const size_t outSize,
const double scalar) :
inSize(0),
outSize(outSize)
{
constantOutput = OutputDataType(outSize, 1);
constantOutput.fill(scalar);
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
void Constant<InputDataType, OutputDataType>::Forward(
const InputType& input, OutputType& output)
{
if (inSize == 0)
{
inSize = input.n_elem;
}
output = constantOutput;
}
template<typename InputDataType, typename OutputDataType>
template<typename DataType>
void Constant<InputDataType, OutputDataType>::Backward(
const DataType& /* input */, const DataType& /* gy */, DataType& g)
{
g = arma::zeros<DataType>(inSize, 1);
}
template<typename InputDataType, typename OutputDataType>
template<typename Archive>
void Constant<InputDataType, OutputDataType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(CEREAL_NVP(constantOutput));
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/convolution.hpp
+114 -175
View File
@@ -20,7 +20,7 @@
#include <mlpack/methods/ann/convolution_rules/svd_convolution.hpp>
#include <mlpack/core/util/to_lower.hpp>
#include "layer_types.hpp"
#include "layer.hpp"
#include "padding.hpp"
namespace mlpack {
@@ -30,7 +30,7 @@ namespace ann /** Artificial Neural Network. */ {
* Implementation of the Convolution class. The Convolution class represents a
* single layer of a neural network.
* Example usage:
*
*
* Suppose we want to pass a matrix M (2744x100) to a `Convolution` layer;
* in this example, `M` was obtained from "flattening" 100 images (or Mel
* cepstral coefficients, if we talk about speech, or whatever you like) of
@@ -62,100 +62,95 @@ namespace ann /** Artificial Neural Network. */ {
* @tparam ForwardConvolutionRule Convolution to perform forward process.
* @tparam BackwardConvolutionRule Convolution to perform backward process.
* @tparam GradientConvolutionRule Convolution to calculate gradient.
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template <
typename ForwardConvolutionRule = NaiveConvolution<ValidConvolution>,
typename BackwardConvolutionRule = NaiveConvolution<FullConvolution>,
typename GradientConvolutionRule = NaiveConvolution<ValidConvolution>,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
typename MatType = arma::mat
>
class Convolution
class ConvolutionType : public Layer<MatType>
{
public:
//! Create the Convolution object.
Convolution();
//! Create the ConvolutionType object.
ConvolutionType();
/**
* Create the Convolution object using the specified number of input maps,
* output maps, filter size, stride and padding parameter.
* Create the ConvolutionType object using the specified number of output
* maps, filter size, stride and padding parameter.
*
* @param inSize The number of input maps.
* @param outSize The number of output maps.
* @param maps The number of output maps.
* @param kernelWidth Width of the filter/kernel.
* @param kernelHeight Height of the filter/kernel.
* @param strideWidth Stride of filter application in the x direction.
* @param strideHeight Stride of filter application in the y direction.
* @param padW Padding width of the input.
* @param padH Padding height of the input.
* @param inputWidth The width of the input data.
* @param inputHeight The height of the input data.
* @param paddingType The type of padding (Valid or Same). Defaults to None.
* @param paddingType The type of padding ("valid" or "same"). Defaults to
* "none". If not specified or "none", the values for `padW` and `padH`
* will be used.
*/
Convolution(const size_t inSize,
const size_t outSize,
const size_t kernelWidth,
const size_t kernelHeight,
const size_t strideWidth = 1,
const size_t strideHeight = 1,
const size_t padW = 0,
const size_t padH = 0,
const size_t inputWidth = 0,
const size_t inputHeight = 0,
const std::string& paddingType = "None");
ConvolutionType(const size_t maps,
const size_t kernelWidth,
const size_t kernelHeight,
const size_t strideWidth = 1,
const size_t strideHeight = 1,
const size_t padW = 0,
const size_t padH = 0,
const std::string& paddingType = "none");
/**
* Create the Convolution object using the specified number of input maps,
* output maps, filter size, stride and padding parameter.
*
* @param inSize The number of input maps.
* @param outSize The number of output maps.
* @param maps The number of output maps.
* @param kernelWidth Width of the filter/kernel.
* @param kernelHeight Height of the filter/kernel.
* @param strideWidth Stride of filter application in the x direction.
* @param strideHeight Stride of filter application in the y direction.
* @param padW A two-value tuple indicating padding widths of the input.
* First value is padding at left side. Second value is padding on
* right side.
* @param padH A two-value tuple indicating padding heights of the input.
* First value is padding at top. Second value is padding on
* bottom.
* @param inputWidth The width of the input data.
* @param inputHeight The height of the input data.
* @param paddingType The type of padding (Valid or Same). Defaults to None.
* @param padW A two-value tuple indicating padding widths of the input. The
* first value is the padding for the left side; the second value is the
* padding on the right side.
* @param padH A two-value tuple indicating padding heights of the input. The
* first value is the padding for the top; the second value is the
* padding on the bottom.
* @param paddingType The type of padding ("valid" or "same"). Defaults to
* "none". If not specified or "none", the values for `padW` and `padH`
* will be used.
*/
Convolution(const size_t inSize,
const size_t outSize,
const size_t kernelWidth,
const size_t kernelHeight,
const size_t strideWidth,
const size_t strideHeight,
const std::tuple<size_t, size_t>& padW,
const std::tuple<size_t, size_t>& padH,
const size_t inputWidth = 0,
const size_t inputHeight = 0,
const std::string& paddingType = "None");
ConvolutionType(const size_t maps,
const size_t kernelWidth,
const size_t kernelHeight,
const size_t strideWidth,
const size_t strideHeight,
const std::tuple<size_t, size_t>& padW,
const std::tuple<size_t, size_t>& padH,
const std::string& paddingType = "none");
//! Copy constructor.
Convolution(const Convolution& layer);
//! Clone the ConvolutionType object. This handles polymorphism correctly.
ConvolutionType* Clone() const { return new ConvolutionType(*this); }
//! Move constructor.
Convolution(Convolution&&);
//! Copy the given ConvolutionType (but not weights).
ConvolutionType(const ConvolutionType& layer);
//! Copy assignment operator.
Convolution& operator=(const Convolution& layer);
//! Take ownership of the given ConvolutionType (but not weights).
ConvolutionType(ConvolutionType&&);
//! Move assignment operator.
Convolution& operator=(Convolution&& layer);
//! Copy the given ConvolutionType (but not weights).
ConvolutionType& operator=(const ConvolutionType& layer);
//! Take ownership of the given ConvolutionType (but not weights).
ConvolutionType& operator=(ConvolutionType&& layer);
// Virtual destructor.
virtual ~ConvolutionType() { }
/*
* Set the weight and bias term.
*/
void Reset();
void SetWeights(typename MatType::elem_type* weightsPtr);
/**
* Ordinary feed forward pass of a neural network, evaluating the function
@@ -164,8 +159,7 @@ class Convolution
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of a neural network, calculating the function
@@ -176,135 +170,91 @@ class Convolution
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
void Backward(const MatType& /* input */,
const MatType& gy,
MatType& g);
/*
/**
* Calculate the gradient using the output delta and the input activation.
*
* @param input The input parameter used for calculating the gradient.
* @param error The calculated error.
* @param gradient The calculated gradient.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient);
void Gradient(const MatType& /* input */,
const MatType& error,
MatType& gradient);
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
MatType const& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
MatType& Parameters() { return weights; }
//! Get the weight of the layer.
arma::cube const& Weight() const { return weight; }
//! Modify the weight of the layer.
arma::cube& Weight() { return weight; }
//! Get the weight of the layer as a cube.
arma::Cube<typename MatType::elem_type> const& Weight() const
{
return weight;
}
//! Modify the weight of the layer as a cube.
arma::Cube<typename MatType::elem_type>& Weight() { return weight; }
//! Get the bias of the layer.
arma::mat const& Bias() const { return bias; }
MatType const& Bias() const { return bias; }
//! Modify the bias of the layer.
arma::mat& Bias() { return bias; }
//! Get the input parameter.
InputDataType const& InputParameter() const { return inputParameter; }
//! Modify the input parameter.
InputDataType& InputParameter() { return inputParameter; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the gradient.
OutputDataType const& Gradient() const { return gradient; }
//! Modify the gradient.
OutputDataType& Gradient() { return gradient; }
//! Get the input width.
size_t InputWidth() const { return inputWidth; }
//! Modify input the width.
size_t& InputWidth() { return inputWidth; }
//! Get the input height.
size_t InputHeight() const { return inputHeight; }
//! Modify the input height.
size_t& InputHeight() { return inputHeight; }
//! Get the output width.
size_t OutputWidth() const { return outputWidth; }
//! Modify the output width.
size_t& OutputWidth() { return outputWidth; }
//! Get the output height.
size_t OutputHeight() const { return outputHeight; }
//! Modify the output height.
size_t& OutputHeight() { return outputHeight; }
//! Get the number of input maps.
size_t InputSize() const { return inSize; }
MatType& Bias() { return bias; }
//! Get the number of output maps.
size_t OutputSize() const { return outSize; }
size_t const& Maps() const { return maps; }
//! Get the kernel width.
size_t KernelWidth() const { return kernelWidth; }
size_t const& KernelWidth() const { return kernelWidth; }
//! Modify the kernel width.
size_t& KernelWidth() { return kernelWidth; }
//! Get the kernel height.
size_t KernelHeight() const { return kernelHeight; }
size_t const& KernelHeight() const { return kernelHeight; }
//! Modify the kernel height.
size_t& KernelHeight() { return kernelHeight; }
//! Get the stride width.
size_t StrideWidth() const { return strideWidth; }
size_t const& StrideWidth() const { return strideWidth; }
//! Modify the stride width.
size_t& StrideWidth() { return strideWidth; }
//! Get the stride height.
size_t StrideHeight() const { return strideHeight; }
size_t const& StrideHeight() const { return strideHeight; }
//! Modify the stride height.
size_t& StrideHeight() { return strideHeight; }
//! Get the top padding height.
size_t PadHTop() const { return padHTop; }
size_t const& PadHTop() const { return padHTop; }
//! Modify the top padding height.
size_t& PadHTop() { return padHTop; }
//! Get the bottom padding height.
size_t PadHBottom() const { return padHBottom; }
size_t const& PadHBottom() const { return padHBottom; }
//! Modify the bottom padding height.
size_t& PadHBottom() { return padHBottom; }
//! Get the left padding width.
size_t PadWLeft() const { return padWLeft; }
size_t const& PadWLeft() const { return padWLeft; }
//! Modify the left padding width.
size_t& PadWLeft() { return padWLeft; }
//! Get the right padding width.
size_t PadWRight() const { return padWRight; }
size_t const& PadWRight() const { return padWRight; }
//! Modify the right padding width.
size_t& PadWRight() { return padWRight; }
//! Get size of weights for the layer.
size_t WeightSize() const
{
return (outSize * inSize * kernelWidth * kernelHeight) + outSize;
return (maps * inMaps * higherInDimensions * kernelWidth * kernelHeight) +
maps;
}
//! Get the shape of the input.
size_t InputShape() const
{
return inputHeight * inputWidth * inSize;
}
//! Compute the output dimensions of the layer based on `InputDimensions()`.
void ComputeOutputDimensions();
/**
* Serialize the layer.
@@ -313,7 +263,7 @@ class Convolution
void serialize(Archive& ar, const uint32_t /* version */);
private:
/*
/**
* Return the convolution output size.
*
* @param size The size of the input (row or column).
@@ -332,12 +282,12 @@ class Convolution
return std::floor(size + pSideOne + pSideTwo - k) / s + 1;
}
/*
/**
* Function to assign padding such that output size is same as input size.
*/
void InitializeSamePadding();
/*
/**
* Rotates a 3rd-order tensor counterclockwise by 180 degrees.
*
* @param input The input data to be rotated.
@@ -353,7 +303,7 @@ class Convolution
output.slice(s) = arma::fliplr(arma::flipud(input.slice(s)));
}
/*
/**
* Rotates a dense matrix counterclockwise by 180 degrees.
*
* @param input The input data to be rotated.
@@ -366,11 +316,8 @@ class Convolution
output = arma::fliplr(arma::flipud(input));
}
//! Locally-stored number of input channels.
size_t inSize;
//! Locally-stored number of output channels.
size_t outSize;
size_t maps;
//! Locally-stored number of input units.
size_t batchSize;
@@ -400,54 +347,46 @@ class Convolution
size_t padHTop;
//! Locally-stored weight object.
OutputDataType weights;
MatType weights;
//! Locally-stored weight object.
arma::cube weight;
arma::Cube<typename MatType::elem_type> weight;
//! Locally-stored bias term object.
arma::mat bias;
//! Locally-stored input width.
size_t inputWidth;
//! Locally-stored input height.
size_t inputHeight;
//! Locally-stored output width.
size_t outputWidth;
//! Locally-stored output height.
size_t outputHeight;
MatType bias;
//! Locally-stored transformed output parameter.
arma::cube outputTemp;
arma::Cube<typename MatType::elem_type> outputTemp;
//! Locally-stored transformed padded input parameter.
arma::cube inputPaddedTemp;
MatType inputPadded;
//! Locally-stored transformed error parameter.
arma::cube gTemp;
arma::Cube<typename MatType::elem_type> gTemp;
//! Locally-stored transformed gradient parameter.
arma::cube gradientTemp;
arma::Cube<typename MatType::elem_type> gradientTemp;
//! Locally-stored padding layer.
ann::Padding<> padding;
ann::Padding padding;
//! Locally-stored delta object.
OutputDataType delta;
//! Type of padding.
std::string paddingType;
//! Locally-stored gradient object.
OutputDataType gradient;
//! Locally-stored input parameter object.
InputDataType inputParameter;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
//! Locally-cached number of input maps.
size_t inMaps;
//! Locally-cached higher-order input dimensions.
size_t higherInDimensions;
}; // class Convolution
// Standard Convolution layer.
typedef ConvolutionType<
NaiveConvolution<ValidConvolution>,
NaiveConvolution<FullConvolution>,
NaiveConvolution<ValidConvolution>,
arma::mat
> Convolution;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/convolution_impl.hpp
+473 -458
View File
File diff suppressed because it is too large Load Diff
src/mlpack/methods/ann/layer/dropconnect.hpp
+53 -96
View File
@@ -16,10 +16,7 @@
#include <mlpack/prereqs.hpp>
#include "layer_types.hpp"
#include "add_merge.hpp"
#include "linear.hpp"
#include "sequential.hpp"
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -28,55 +25,58 @@ namespace ann /** Artificial Neural Network. */ {
* The DropConnect layer is a regularizer that randomly with probability
* ratio sets the connection values to zero and scales the remaining
* elements by factor 1 /(1 - ratio). The output is scaled with 1 / (1 - p)
* when deterministic is false. In the deterministic mode(during testing),
* the layer just computes the output. The output is computed according
* to the input layer. If no input layer is given, it will take a linear layer
* as default.
* when in training mode. During testing, the layer just computes the output.
* The output is computed according to the input layer. If no input layer is
* given, it will take a linear layer as default.
*
* Note:
* During training you should set deterministic to false and during testing
* you should set deterministic to true.
*
* For more information, see the following.
* For more information, see the following.
*
* @code
* @inproceedings{WanICML2013,
* title={Regularization of Neural Networks using DropConnect},
* title = {Regularization of Neural Networks using DropConnect},
* booktitle = {Proceedings of the 30th International Conference on Machine
* Learning(ICML - 13)},
* author = {Li Wan and Matthew Zeiler and Sixin Zhang and Yann L. Cun and
* Rob Fergus},
* year = {2013},
* url = {http://proceedings.mlr.press/v28/wan13.pdf}
* author = {Li Wan and Matthew Zeiler and Sixin Zhang and Yann L. Cun and
* Rob Fergus},
* year = {2013},
* url = {http://proceedings.mlr.press/v28/wan13.pdf}
* }
* @endcode
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template<
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class DropConnect
template<typename MatType = arma::mat>
class DropConnectType : public Layer<MatType>
{
public:
//! Create the DropConnect object.
DropConnect();
DropConnectType();
/**
* Creates the DropConnect Layer as a Linear Object that takes input size,
* output size and ratio as parameter.
* Creates the DropConnect Layer as a Linear Object that takes the number of
* output units and a ratio as parameter.
*
* @param inSize The number of input units.
* @param outSize The number of output units.
* @param ratio The probability of setting a value to zero.
*/
DropConnect(const size_t inSize,
const size_t outSize,
const double ratio = 0.5);
DropConnectType(const size_t outSize,
const double ratio = 0.5);
//! Clone the DropConnectType object. This handles polymorphism correctly.
DropConnectType* Clone() const { return new DropConnectType(*this); }
// Virtual destructor.
virtual ~DropConnectType();
//! Copy the given DropConnectType (except for weights).
DropConnectType(const DropConnectType& other);
//! Take ownership of the given DropConnectType (except for weights).
DropConnectType(DropConnectType&& other);
//! Copy the given DropConnectType (except for weights).
DropConnectType& operator=(const DropConnectType& other);
//! Take ownership of the given DropConnectType (except for weights).
DropConnectType& operator=(DropConnectType&& other);
/**
* Ordinary feed forward pass of the DropConnect layer.
@@ -84,8 +84,7 @@ class DropConnect
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of the DropConnect layer.
@@ -94,10 +93,7 @@ class DropConnect
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& input,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
void Backward(const MatType& input, const MatType& gy, MatType& g);
/**
* Calculate the gradient using the output delta and the input activation.
@@ -106,39 +102,7 @@ class DropConnect
* @param error The calculated error.
* @param * (gradient) The calculated gradient.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& /* gradient */);
//! Get the model modules.
std::vector<LayerTypes<> >& Model() { return network; }
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the gradient.
OutputDataType const& Gradient() const { return gradient; }
//! Modify the gradient.
OutputDataType& Gradient() { return gradient; }
//! The value of the deterministic parameter.
bool Deterministic() const { return deterministic; }
//! Modify the value of the deterministic parameter.
bool &Deterministic() { return deterministic; }
void Gradient(const MatType& input, const MatType& error, MatType& gradient);
//! The probability of setting a value to zero.
double Ratio() const { return ratio; }
@@ -150,8 +114,14 @@ class DropConnect
scale = 1.0 / (1.0 - ratio);
}
//! Return the size of the weight matrix.
size_t WeightSize() const { return 0; }
//! Compute the output dimensions of the layer based on `InputDimensions()`.
void ComputeOutputDimensions();
//! Return the size of the weights.
size_t WeightSize() const { return baseLayer->WeightSize(); }
// Set the weights to use the given memory `weightsPtr`.
void SetWeights(typename MatType::elem_type* weightsPtr);
/**
* Serialize the layer.
@@ -166,34 +136,21 @@ class DropConnect
//! The scale fraction.
double scale;
//! Locally-stored weight object.
OutputDataType weights;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored gradient object.
OutputDataType gradient;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
//! Locally-stored mask object.
OutputDataType mask;
//! If true dropout and scaling is disabled, see notes above.
bool deterministic;
MatType mask;
//! Denoise mask for the weights.
OutputDataType denoise;
MatType denoise;
//! Locally-stored layer module.
LayerTypes<> baseLayer;
//! Locally-stored network modules.
std::vector<LayerTypes<> > network;
Layer<MatType>* baseLayer;
}; // class DropConnect.
// Convenience typedefs.
// Standard DropConnect layer.
typedef DropConnectType<arma::mat> DropConnect;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/dropconnect_impl.hpp
+107 -59
View File
@@ -17,112 +17,160 @@
// In case it hasn't yet been included.
#include "dropconnect.hpp"
#include "../visitor/delete_visitor.hpp"
#include "../visitor/forward_visitor.hpp"
#include "../visitor/backward_visitor.hpp"
#include "../visitor/gradient_visitor.hpp"
#include "../visitor/parameters_set_visitor.hpp"
#include "../visitor/parameters_visitor.hpp"
#include "linear.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
DropConnect<InputDataType, OutputDataType>::DropConnect() :
template<typename MatType>
DropConnectType<MatType>::DropConnectType() :
Layer<MatType>(),
ratio(0.5),
scale(2.0),
deterministic(true)
baseLayer(new LinearType<MatType>(0))
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
DropConnect<InputDataType, OutputDataType>::DropConnect(
const size_t inSize,
template<typename MatType>
DropConnectType<MatType>::DropConnectType(
const size_t outSize,
const double ratio) :
Layer<MatType>(),
ratio(ratio),
scale(1.0 / (1 - ratio)),
baseLayer(new Linear<InputDataType, OutputDataType>(inSize, outSize))
baseLayer(new LinearType<MatType>(outSize))
{
network.push_back(baseLayer);
// Nothing to do.
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void DropConnect<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input,
arma::Mat<eT>& output)
template<typename MatType>
DropConnectType<MatType>::~DropConnectType()
{
// The DropConnect mask will not be multiplied in the deterministic mode
// (during testing).
if (deterministic)
delete baseLayer;
}
template<typename MatType>
DropConnectType<MatType>::DropConnectType(const DropConnectType& other) :
Layer<MatType>(other),
ratio(other.ratio),
scale(other.scale),
baseLayer(other.baseLayer->Clone())
{
// Nothing to do.
}
template<typename MatType>
DropConnectType<MatType>::DropConnectType(DropConnectType&& other) :
Layer<MatType>(std::move(other)),
ratio(std::move(other.ratio)),
scale(std::move(other.scale)),
baseLayer(std::move(other.baseLayer))
{
// Nothing to do.
}
template<typename MatType>
DropConnectType<MatType>&
DropConnectType<MatType>::operator=(const DropConnectType& other)
{
if (&other != this)
{
boost::apply_visitor(ForwardVisitor(input, output), baseLayer);
Layer<MatType>::operator=(other);
ratio = other.ratio;
scale = other.scale;
baseLayer = other.baseLayer->Clone();
}
return *this;
}
template<typename MatType>
DropConnectType<MatType>&
DropConnectType<MatType>::operator=(DropConnectType&& other)
{
if (&other != this)
{
Layer<MatType>::operator=(std::move(other));
ratio = std::move(other.ratio);
scale = std::move(other.scale);
baseLayer = std::move(other.baseLayer);
}
return *this;
}
template<typename MatType>
void DropConnectType<MatType>::Forward(const MatType& input, MatType& output)
{
// The DropConnect mask will not be multiplied in testing mode.
if (!this->training)
{
baseLayer->Forward(input, output);
}
else
{
// Save weights for denoising.
boost::apply_visitor(ParametersVisitor(denoise), baseLayer);
denoise = baseLayer->Parameters();
// Scale with input / (1 - ratio) and set values to zero with
// probability ratio.
mask = arma::randu<arma::Mat<eT> >(denoise.n_rows, denoise.n_cols);
mask = arma::randu<MatType>(denoise.n_rows, denoise.n_cols);
mask.transform([&](double val) { return (val > ratio); });
arma::mat tmp = denoise % mask;
boost::apply_visitor(ParametersSetVisitor(tmp), baseLayer);
boost::apply_visitor(ForwardVisitor(input, output), baseLayer);
baseLayer->Parameters() = denoise % mask;
baseLayer->Forward(input, output);
output = output * scale;
}
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void DropConnect<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& input,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g)
template<typename MatType>
void DropConnectType<MatType>::Backward(
const MatType& input,
const MatType& gy,
MatType& g)
{
boost::apply_visitor(BackwardVisitor(input, gy, g), baseLayer);
baseLayer->Backward(input, gy, g);
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void DropConnect<InputDataType, OutputDataType>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& /* gradient */)
template<typename MatType>
void DropConnectType<MatType>::Gradient(
const MatType& input,
const MatType& error,
MatType& gradient)
{
boost::apply_visitor(GradientVisitor(input, error),
baseLayer);
baseLayer->Gradient(input, error, gradient);
// Denoise the weights.
boost::apply_visitor(ParametersSetVisitor(denoise), baseLayer);
baseLayer->Parameters() = denoise;
}
template<typename InputDataType, typename OutputDataType>
template<typename MatType>
void DropConnectType<MatType>::ComputeOutputDimensions()
{
// Propagate input dimensions to the base layer.
baseLayer->InputDimensions() = this->inputDimensions;
this->outputDimensions = baseLayer->OutputDimensions();
}
template<typename MatType>
void DropConnectType<MatType>::SetWeights(
typename MatType::elem_type* weightsPtr)
{
baseLayer->SetWeights(weightsPtr);
}
template<typename MatType>
template<typename Archive>
void DropConnect<InputDataType, OutputDataType>::serialize(
void DropConnectType<MatType>::serialize(
Archive& ar, const uint32_t /* version */)
{
// Delete the old network first, if needed.
if (cereal::is_loading<Archive>())
{
boost::apply_visitor(DeleteVisitor(), baseLayer);
}
ar(cereal::base_class<Layer<MatType>>(this));
ar(CEREAL_NVP(ratio));
ar(CEREAL_NVP(scale));
ar(CEREAL_VARIANT_POINTER(baseLayer));
if (cereal::is_loading<Archive>())
{
network.clear();
network.push_back(baseLayer);
}
ar(CEREAL_POINTER(baseLayer));
}
} // namespace ann
src/mlpack/methods/ann/layer/dropout.hpp
+29 -54
View File
@@ -15,18 +15,16 @@
#include <mlpack/prereqs.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
/**
* The dropout layer is a regularizer that randomly with probability 'ratio'
* sets input values to zero and scales the remaining elements by factor 1 /
* (1 - ratio) rather than during test time so as to keep the expected sum same.
* In the deterministic mode (during testing), there is no change in the input.
*
* Note: During training you should set deterministic to false and during
* testing you should set deterministic to true.
* When the layer is in testing mode, there is no change in the input.
*
* For more information, see the following.
*
@@ -43,14 +41,11 @@ namespace ann /** Artificial Neural Network. */ {
* }
* @endcode
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template<typename InputDataType = arma::mat,
typename OutputDataType = arma::mat>
class Dropout
template<typename MatType = arma::mat>
class DropoutType : public Layer<MatType>
{
public:
/**
@@ -58,19 +53,22 @@ class Dropout
*
* @param ratio The probability of setting a value to zero.
*/
Dropout(const double ratio = 0.5);
DropoutType(const double ratio = 0.5);
//! Copy Constructor
Dropout(const Dropout& layer);
//! Clone the DropoutType object. This handles polymorphism correctly.
DropoutType* Clone() const { return new DropoutType(*this); }
//! Move Constructor
Dropout(const Dropout&&);
// Virtual destructor.
virtual ~DropoutType() { }
//! Copy assignment operator
Dropout& operator=(const Dropout& layer);
//! Move assignment operator
Dropout& operator=(Dropout&& layer);
//! Copy the given DropoutType.
DropoutType(const DropoutType& other);
//! Take ownership of the given DropoutType.
DropoutType(DropoutType&& other);
//! Copy the given DropoutType.
DropoutType& operator=(const DropoutType& other);
//! Take ownership of the given DropoutType.
DropoutType& operator=(DropoutType&& other);
/**
* Ordinary feed forward pass of the dropout layer.
@@ -78,8 +76,7 @@ class Dropout
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of the dropout layer.
@@ -88,25 +85,7 @@ class Dropout
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the detla.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! The value of the deterministic parameter.
bool Deterministic() const { return deterministic; }
//! Modify the value of the deterministic parameter.
bool& Deterministic() { return deterministic; }
void Backward(const MatType& /* input */, const MatType& gy, MatType& g);
//! The probability of setting a value to zero.
double Ratio() const { return ratio; }
@@ -125,24 +104,20 @@ class Dropout
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
//! Locally-stored mast object.
OutputDataType mask;
//! Locally-stored mask object.
MatType mask;
//! The probability of setting a value to zero.
double ratio;
//! The scale fraction.
double scale;
}; // class DropoutType
//! If true dropout and scaling is disabled, see notes above.
bool deterministic;
}; // class Dropout
// Convenience typedefs.
// Standard Dropout layer.
typedef DropoutType<arma::mat> Dropout;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/dropout_impl.hpp
+45 -51
View File
@@ -19,73 +19,66 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
Dropout<InputDataType, OutputDataType>::Dropout(
template<typename MatType>
DropoutType<MatType>::DropoutType(
const double ratio) :
ratio(ratio),
scale(1.0 / (1.0 - ratio)),
deterministic(false)
scale(1.0 / (1.0 - ratio))
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
Dropout<InputDataType, OutputDataType>::Dropout(
const Dropout& layer) :
ratio(layer.ratio),
scale(layer.scale),
deterministic(layer.deterministic)
template<typename MatType>
DropoutType<MatType>::DropoutType(const DropoutType& other) :
Layer<MatType>(other),
ratio(other.ratio),
scale(other.scale)
{
// Nothing to do here.
// Nothing to do.
}
template<typename InputDataType, typename OutputDataType>
Dropout<InputDataType, OutputDataType>::Dropout(
const Dropout&& layer) :
ratio(std::move(layer.ratio)),
scale(std::move(scale)),
deterministic(std::move(deterministic))
template<typename MatType>
DropoutType<MatType>::DropoutType(DropoutType&& other) :
Layer<MatType>(std::move(other)),
ratio(std::move(other.ratio)),
scale(std::move(other.scale))
{
// Nothing to do here.
// Nothing to do.
}
template<typename InputDataType, typename OutputDataType>
Dropout<InputDataType, OutputDataType>&
Dropout<InputDataType, OutputDataType>::
operator=(const Dropout& layer)
template<typename MatType>
DropoutType<MatType>&
DropoutType<MatType>::operator=(const DropoutType& other)
{
if (this != &layer)
if (&other != this)
{
ratio = layer.ratio;
scale = layer.scale;
deterministic = layer.deterministic;
Layer<MatType>::operator=(other);
ratio = other.ratio;
scale = other.scale;
}
return *this;
}
template<typename InputDataType, typename OutputDataType>
Dropout<InputDataType, OutputDataType>&
Dropout<InputDataType, OutputDataType>::
operator=(Dropout&& layer)
template<typename MatType>
DropoutType<MatType>&
DropoutType<MatType>::operator=(DropoutType&& other)
{
if (this != &layer)
if (&other != this)
{
ratio = std::move(layer.ratio);
scale = std::move(layer.scale);
deterministic = std::move(layer.deterministic);
Layer<MatType>::operator=(std::move(other));
ratio = std::move(other.ratio);
scale = std::move(other.scale);
}
return *this;
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void Dropout<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input,
arma::Mat<eT>& output)
template<typename MatType>
void DropoutType<MatType>::Forward(const MatType& input, MatType& output)
{
// The dropout mask will not be multiplied in the deterministic mode
// (during testing).
if (deterministic)
// The dropout mask will not be multiplied in testing mode.
if (!this->training)
{
output = input;
}
@@ -93,28 +86,29 @@ void Dropout<InputDataType, OutputDataType>::Forward(
{
// Scale with input / (1 - ratio) and set values to zero with probability
// 'ratio'.
mask = arma::randu<arma::Mat<eT> >(input.n_rows, input.n_cols);
mask = arma::randu<MatType>(input.n_rows, input.n_cols);
mask.transform([&](double val) { return (val > ratio); });
output = input % mask * scale;
}
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void Dropout<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g)
template<typename MatType>
void DropoutType<MatType>::Backward(
const MatType& /* input */,
const MatType& gy,
MatType& g)
{
g = gy % mask * scale;
}
template<typename InputDataType, typename OutputDataType>
template<typename MatType>
template<typename Archive>
void Dropout<InputDataType, OutputDataType>::serialize(
void DropoutType<MatType>::serialize(
Archive& ar,
const uint32_t /* version */)
{
ar(cereal::base_class<Layer<MatType>>(this));
ar(CEREAL_NVP(ratio));
// Reset scale.
src/mlpack/methods/ann/layer/gru_impl.hpp
-411
View File
@@ -1,411 +0,0 @@
/**
* @file methods/ann/layer/gru_impl.hpp
* @author Sumedh Ghaisas
*
* Implementation of the GRU class, which implements a gru network
* layer.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_GRU_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_GRU_IMPL_HPP
// In case it hasn't yet been included.
#include "gru.hpp"
#include "../visitor/forward_visitor.hpp"
#include "../visitor/backward_visitor.hpp"
#include "../visitor/gradient_visitor.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
GRU<InputDataType, OutputDataType>::GRU()
{
// Nothing to do here.
}
template <typename InputDataType, typename OutputDataType>
GRU<InputDataType, OutputDataType>::GRU(
const size_t inSize,
const size_t outSize,
const size_t rho) :
inSize(inSize),
outSize(outSize),
rho(rho),
batchSize(1),
forwardStep(0),
backwardStep(0),
gradientStep(0),
deterministic(false)
{
// Input specific linear layers(for zt, rt, ot).
input2GateModule = new Linear<>(inSize, 3 * outSize);
// Previous output gates (for zt and rt).
output2GateModule = new LinearNoBias<>(outSize, 2 * outSize);
// Previous output gate for ot.
outputHidden2GateModule = new LinearNoBias<>(outSize, outSize);
network.push_back(input2GateModule);
network.push_back(output2GateModule);
network.push_back(outputHidden2GateModule);
inputGateModule = new SigmoidLayer<>();
forgetGateModule = new SigmoidLayer<>();
hiddenStateModule = new TanHLayer<>();
network.push_back(inputGateModule);
network.push_back(hiddenStateModule);
network.push_back(forgetGateModule);
prevError = arma::zeros<arma::mat>(3 * outSize, batchSize);
allZeros = arma::zeros<arma::mat>(outSize, batchSize);
outParameter.emplace_back(allZeros.memptr(),
allZeros.n_rows, allZeros.n_cols, false, true);
prevOutput = outParameter.begin();
backIterator = outParameter.end();
gradIterator = outParameter.end();
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void GRU<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
{
if (input.n_cols != batchSize)
{
batchSize = input.n_cols;
prevError.resize(3 * outSize, batchSize);
allZeros.zeros(outSize, batchSize);
// Batch size better not change during an iteration...
if (outParameter.size() > 1)
{
Log::Fatal << "GRU<>::Forward(): batch size cannot change during a "
<< "forward pass!" << std::endl;
}
outParameter.clear();
outParameter.emplace_back(allZeros.memptr(),
allZeros.n_rows, allZeros.n_cols, false, true);
prevOutput = outParameter.begin();
backIterator = outParameter.end();
gradIterator = outParameter.end();
}
// Process the input linearly(zt, rt, ot).
boost::apply_visitor(ForwardVisitor(input,
boost::apply_visitor(outputParameterVisitor, input2GateModule)),
input2GateModule);
// Process the output(zt, rt) linearly.
boost::apply_visitor(ForwardVisitor(*prevOutput,
boost::apply_visitor(outputParameterVisitor, output2GateModule)),
output2GateModule);
// Merge the outputs(zt and rt).
output = (boost::apply_visitor(outputParameterVisitor,
input2GateModule).submat(0, 0, 2 * outSize - 1, batchSize - 1) +
boost::apply_visitor(outputParameterVisitor, output2GateModule));
// Pass the first outSize through inputGate(it).
boost::apply_visitor(ForwardVisitor(output.submat(
0, 0, 1 * outSize - 1, batchSize - 1), boost::apply_visitor(
outputParameterVisitor, inputGateModule)), inputGateModule);
// Pass the second through forgetGate.
boost::apply_visitor(ForwardVisitor(output.submat(
1 * outSize, 0, 2 * outSize - 1, batchSize - 1),
boost::apply_visitor(outputParameterVisitor, forgetGateModule)),
forgetGateModule);
arma::mat modInput = (boost::apply_visitor(outputParameterVisitor,
forgetGateModule) % *prevOutput);
// Pass that through the outputHidden2GateModule.
boost::apply_visitor(ForwardVisitor(modInput,
boost::apply_visitor(outputParameterVisitor, outputHidden2GateModule)),
outputHidden2GateModule);
// Merge for ot.
arma::mat outputH = boost::apply_visitor(outputParameterVisitor,
input2GateModule).submat(2 * outSize, 0, 3 * outSize - 1, batchSize - 1) +
boost::apply_visitor(outputParameterVisitor, outputHidden2GateModule);
// Pass it through hiddenGate.
boost::apply_visitor(ForwardVisitor(outputH,
boost::apply_visitor(outputParameterVisitor, hiddenStateModule)),
hiddenStateModule);
// Update the output (nextOutput): cmul1 + cmul2
// Where cmul1 is input gate * prevOutput and
// cmul2 is (1 - input gate) * hidden gate.
output = (boost::apply_visitor(outputParameterVisitor, inputGateModule)
% (*prevOutput - boost::apply_visitor(outputParameterVisitor,
hiddenStateModule))) + boost::apply_visitor(outputParameterVisitor,
hiddenStateModule);
forwardStep++;
if (forwardStep == rho)
{
forwardStep = 0;
if (!deterministic)
{
outParameter.emplace_back(allZeros.memptr(),
allZeros.n_rows, allZeros.n_cols, false, true);
prevOutput = --outParameter.end();
}
else
{
*prevOutput = arma::mat(allZeros.memptr(),
allZeros.n_rows, allZeros.n_cols, false, true);
}
}
else if (!deterministic)
{
outParameter.push_back(output);
prevOutput = --outParameter.end();
}
else
{
if (forwardStep == 1)
{
outParameter.clear();
outParameter.push_back(output);
prevOutput = outParameter.begin();
}
else
{
*prevOutput = output;
}
}
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void GRU<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& input, const arma::Mat<eT>& gy, arma::Mat<eT>& g)
{
if (input.n_cols != batchSize)
{
batchSize = input.n_cols;
prevError.resize(3 * outSize, batchSize);
allZeros.zeros(outSize, batchSize);
// Batch size better not change during an iteration...
if (outParameter.size() > 1)
{
Log::Fatal << "GRU<>::Forward(): batch size cannot change during a "
<< "forward pass!" << std::endl;
}
outParameter.clear();
outParameter.emplace_back(allZeros.memptr(),
allZeros.n_rows, allZeros.n_cols, false, true);
prevOutput = outParameter.begin();
backIterator = outParameter.end();
gradIterator = outParameter.end();
}
arma::Mat<eT> gyLocal;
if ((outParameter.size() - backwardStep - 1) % rho != 0 && backwardStep != 0)
{
gyLocal = gy + boost::apply_visitor(deltaVisitor, output2GateModule);
}
else
{
gyLocal = arma::Mat<eT>(((arma::Mat<eT>&) gy).memptr(), gy.n_rows,
gy.n_cols, false, false);
}
if (backIterator == outParameter.end())
{
backIterator = --(--outParameter.end());
}
// Delta zt.
arma::mat dZt = gyLocal % (*backIterator -
boost::apply_visitor(outputParameterVisitor,
hiddenStateModule));
// Delta ot.
arma::mat dOt = gyLocal % (arma::ones<arma::mat>(outSize, batchSize) -
boost::apply_visitor(outputParameterVisitor, inputGateModule));
// Delta of input gate.
boost::apply_visitor(BackwardVisitor(boost::apply_visitor(
outputParameterVisitor, inputGateModule), dZt,
boost::apply_visitor(deltaVisitor, inputGateModule)),
inputGateModule);
// Delta of hidden gate.
boost::apply_visitor(BackwardVisitor(boost::apply_visitor(
outputParameterVisitor, hiddenStateModule), dOt,
boost::apply_visitor(deltaVisitor, hiddenStateModule)),
hiddenStateModule);
// Delta of outputHidden2GateModule.
boost::apply_visitor(BackwardVisitor(boost::apply_visitor(
outputParameterVisitor, outputHidden2GateModule),
boost::apply_visitor(deltaVisitor, hiddenStateModule),
boost::apply_visitor(deltaVisitor, outputHidden2GateModule)),
outputHidden2GateModule);
// Delta rt.
arma::mat dRt = boost::apply_visitor(deltaVisitor, outputHidden2GateModule) %
*backIterator;
// Delta of forget gate.
boost::apply_visitor(BackwardVisitor(boost::apply_visitor(
outputParameterVisitor, forgetGateModule), dRt,
boost::apply_visitor(deltaVisitor, forgetGateModule)),
forgetGateModule);
// Put delta zt.
prevError.submat(0, 0, 1 * outSize - 1, batchSize - 1) = boost::apply_visitor(
deltaVisitor, inputGateModule);
// Put delta rt.
prevError.submat(1 * outSize, 0, 2 * outSize - 1, batchSize - 1) =
boost::apply_visitor(deltaVisitor, forgetGateModule);
// Put delta ot.
prevError.submat(2 * outSize, 0, 3 * outSize - 1, batchSize - 1) =
boost::apply_visitor(deltaVisitor, hiddenStateModule);
// Get delta ht - 1 for input gate and forget gate.
arma::mat prevErrorSubview = prevError.submat(0, 0, 2 * outSize - 1,
batchSize - 1);
boost::apply_visitor(BackwardVisitor(boost::apply_visitor(
outputParameterVisitor, input2GateModule),
prevErrorSubview,
boost::apply_visitor(deltaVisitor, output2GateModule)),
output2GateModule);
// Add delta ht - 1 from hidden state.
boost::apply_visitor(deltaVisitor, output2GateModule) +=
boost::apply_visitor(deltaVisitor, outputHidden2GateModule) %
boost::apply_visitor(outputParameterVisitor, forgetGateModule);
// Add delta ht - 1 from ht.
boost::apply_visitor(deltaVisitor, output2GateModule) += gyLocal %
boost::apply_visitor(outputParameterVisitor, inputGateModule);
// Get delta input.
boost::apply_visitor(BackwardVisitor(boost::apply_visitor(
outputParameterVisitor, input2GateModule), prevError,
boost::apply_visitor(deltaVisitor, input2GateModule)),
input2GateModule);
backwardStep++;
backIterator--;
g = boost::apply_visitor(deltaVisitor, input2GateModule);
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void GRU<InputDataType, OutputDataType>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& /* error */,
arma::Mat<eT>& /* gradient */)
{
if (input.n_cols != batchSize)
{
batchSize = input.n_cols;
prevError.resize(3 * outSize, batchSize);
allZeros.zeros(outSize, batchSize);
// Batch size better not change during an iteration...
if (outParameter.size() > 1)
{
Log::Fatal << "GRU<>::Forward(): batch size cannot change during a "
<< "forward pass!" << std::endl;
}
outParameter.clear();
outParameter.emplace_back(allZeros.memptr(),
allZeros.n_rows, allZeros.n_cols, false, true);
prevOutput = outParameter.begin();
backIterator = outParameter.end();
gradIterator = outParameter.end();
}
if (gradIterator == outParameter.end())
{
gradIterator = --(--outParameter.end());
}
boost::apply_visitor(GradientVisitor(input, prevError), input2GateModule);
boost::apply_visitor(GradientVisitor(
*gradIterator,
prevError.submat(0, 0, 2 * outSize - 1, batchSize - 1)),
output2GateModule);
boost::apply_visitor(GradientVisitor(
*gradIterator % boost::apply_visitor(outputParameterVisitor,
forgetGateModule),
prevError.submat(2 * outSize, 0, 3 * outSize - 1, batchSize - 1)),
outputHidden2GateModule);
gradIterator--;
}
template<typename InputDataType, typename OutputDataType>
void GRU<InputDataType, OutputDataType>::ResetCell(const size_t /* size */)
{
outParameter.clear();
outParameter.emplace_back(allZeros.memptr(),
allZeros.n_rows, allZeros.n_cols, false, true);
prevOutput = outParameter.begin();
backIterator = outParameter.end();
gradIterator = outParameter.end();
forwardStep = 0;
backwardStep = 0;
}
template<typename InputDataType, typename OutputDataType>
template<typename Archive>
void GRU<InputDataType, OutputDataType>::serialize(
Archive& ar, const uint32_t /* version */)
{
// If necessary, clean memory from the old model.
if (cereal::is_loading<Archive>())
{
boost::apply_visitor(deleteVisitor, input2GateModule);
boost::apply_visitor(deleteVisitor, output2GateModule);
boost::apply_visitor(deleteVisitor, outputHidden2GateModule);
boost::apply_visitor(deleteVisitor, inputGateModule);
boost::apply_visitor(deleteVisitor, forgetGateModule);
boost::apply_visitor(deleteVisitor, hiddenStateModule);
}
ar(CEREAL_NVP(inSize));
ar(CEREAL_NVP(outSize));
ar(CEREAL_NVP(rho));
ar(CEREAL_VARIANT_POINTER(input2GateModule));
ar(CEREAL_VARIANT_POINTER(output2GateModule));
ar(CEREAL_VARIANT_POINTER(outputHidden2GateModule));
ar(CEREAL_VARIANT_POINTER(inputGateModule));
ar(CEREAL_VARIANT_POINTER(forgetGateModule));
ar(CEREAL_VARIANT_POINTER(hiddenStateModule));
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/highway.hpp
-270
View File
@@ -1,270 +0,0 @@
/**
* @file methods/ann/layer/highway.hpp
* @author Konstantin Sidorov
* @author Saksham Bansal
*
* Definition of the Highway layer.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_HIGHWAY_HPP
#define MLPACK_METHODS_ANN_LAYER_HIGHWAY_HPP
#include <mlpack/prereqs.hpp>
#include "../visitor/delete_visitor.hpp"
#include "../visitor/delta_visitor.hpp"
#include "../visitor/output_height_visitor.hpp"
#include "../visitor/output_parameter_visitor.hpp"
#include "../visitor/output_width_visitor.hpp"
#include "layer_types.hpp"
#include "add_merge.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
/**
* Implementation of the Highway layer. The Highway class can vary its behavior
* between that of feed-forward fully connected network container and that
* of a layer which simply passes its inputs through depending on the transform
* gate. Note that the size of the input and output matrices of this class
* should be equal.
*
* For more information, refer the following paper.
*
* @code
* @article{Srivastava2015,
* author = {Rupesh Kumar Srivastava, Klaus Greff, Jurgen Schmidhuber},
* title = {Training Very Deep Networks},
* journal = {Advances in Neural Information Processing Systems},
* year = {2015},
* url = {https://arxiv.org/abs/1507.06228},
* }
* @endcode
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat,
typename... CustomLayers>
class Highway
{
public:
//! Create the Highway object.
Highway();
/**
* Create the Highway object.
*
* @param inSize The number of input units.
* @param model Expose all the network modules.
*/
Highway(const size_t inSize, const bool model = true);
//! Destroy the Highway object.
~Highway();
/**
* Reset the layer parameter.
*/
void Reset();
/**
* Ordinary feed-forward pass of a neural network, evaluating the function
* f(x) by propagating the activity forward through f.
*
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
/**
* Ordinary feed-backward pass of a neural network, calculating the function
* f(x) by propagating x backwards through f. Using the results from the
* feed-forward pass.
*
* @param * (input) The propagated input activation.
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
/**
* Calculate the gradient using the output delta and the input activation.
*
* @param input The input parameter used for calculating the gradient.
* @param error The calculated error.
* @param gradient The calculated gradient.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient);
/**
* Add a new module to the model.
*
* @param args The layer parameter.
*/
template <class LayerType, class... Args>
void Add(Args... args)
{
network.push_back(new LayerType(args...));
networkOwnerships.push_back(true);
}
/**
* Add a new module to the model.
*
* @param layer The Layer to be added to the model.
*/
void Add(LayerTypes<CustomLayers...> layer)
{
network.push_back(layer);
networkOwnerships.push_back(false);
}
//! Return the modules of the model.
std::vector<LayerTypes<CustomLayers...> >& Model()
{
if (model)
{
return network;
}
return empty;
}
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
//! Get the input parameter.
InputDataType const& InputParameter() const { return inputParameter; }
//! Modify the input parameter.
InputDataType& InputParameter() { return inputParameter; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the gradient.
OutputDataType const& Gradient() const { return gradient; }
//! Modify the gradient.
OutputDataType& Gradient() { return gradient; }
//! Get the number of input units.
size_t InSize() const { return inSize; }
//! Get the shape of the input.
size_t InputShape() const
{
return inSize;
}
/**
* Serialize the layer.
*/
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Locally-stored number of input units.
size_t inSize;
//! Parameter which indicates if the modules should be exposed.
bool model;
//! Indicator if we already initialized the model.
bool reset;
//! Locally-stored network modules.
std::vector<LayerTypes<CustomLayers...> > network;
//! The list of network modules we are responsible for.
std::vector<bool> networkOwnerships;
//! Locally-stored empty list of modules.
std::vector<LayerTypes<CustomLayers...> > empty;
//! Locally-stored weight object.
OutputDataType weights;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored gradient object.
OutputDataType gradient;
//! Weights for transformation of output.
OutputDataType transformWeight;
//! Bias for transformation of output.
OutputDataType transformBias;
//! Locally-stored transform gate parameters.
OutputDataType transformGate;
//! Locally-stored transform gate activation.
OutputDataType transformGateActivation;
//! Locally-stored transform gate error.
OutputDataType transformGateError;
//! Locally-stored input parameter object.
InputDataType inputParameter;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
//! The input width.
size_t width;
//! The input height.
size_t height;
//! The normal output without highway network.
OutputDataType networkOutput;
//! Locally-stored delta visitor.
DeltaVisitor deltaVisitor;
//! Locally-stored output parameter visitor.
OutputParameterVisitor outputParameterVisitor;
//! Locally-stored delete visitor.
DeleteVisitor deleteVisitor;
//! Locally-stored output width visitor.
OutputWidthVisitor outputWidthVisitor;
//! Locally-stored output height visitor.
OutputHeightVisitor outputHeightVisitor;
}; // class Highway
} // namespace ann
} // namespace mlpack
// Include implementation.
#include "highway_impl.hpp"
#endif
src/mlpack/methods/ann/layer/highway_impl.hpp
-238
View File
@@ -1,238 +0,0 @@
/**
* @file methods/ann/layer/highway_impl.hpp
* @author Konstantin Sidorov
* @author Saksham Bansal
*
* Implementation of Highway layer.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_HIGHWAY_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_HIGHWAY_IMPL_HPP
// In case it hasn't yet been included.
#include "highway.hpp"
#include "../visitor/forward_visitor.hpp"
#include "../visitor/backward_visitor.hpp"
#include "../visitor/gradient_visitor.hpp"
#include "../visitor/set_input_height_visitor.hpp"
#include "../visitor/set_input_width_visitor.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
Highway<InputDataType, OutputDataType, CustomLayers...>::Highway() :
inSize(0),
model(true),
reset(false),
width(0),
height(0)
{
// Nothing to do here.
}
template<
typename InputDataType, typename OutputDataType, typename... CustomLayers>
Highway<InputDataType, OutputDataType, CustomLayers...>::Highway(
const size_t inSize,
const bool model) :
inSize(inSize),
model(model),
reset(false),
width(0),
height(0)
{
weights.set_size(inSize * inSize + inSize, 1);
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
Highway<InputDataType, OutputDataType, CustomLayers...>::~Highway()
{
if (!model)
{
for (size_t i = 0; i < network.size(); ++i)
{
if (networkOwnerships[i])
boost::apply_visitor(deleteVisitor, network[i]);
}
}
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
void Highway<InputDataType, OutputDataType, CustomLayers...>::Reset()
{
transformWeight = arma::mat(weights.memptr(), inSize, inSize, false, false);
transformBias = arma::mat(weights.memptr() + transformWeight.n_elem,
inSize, 1, false, false);
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void Highway<InputDataType, OutputDataType, CustomLayers...>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
{
boost::apply_visitor(ForwardVisitor(input,
boost::apply_visitor(outputParameterVisitor, network.front())),
network.front());
if (!reset)
{
if (boost::apply_visitor(outputWidthVisitor, network.front()) != 0)
{
width = boost::apply_visitor(outputWidthVisitor, network.front());
}
if (boost::apply_visitor(outputHeightVisitor, network.front()) != 0)
{
height = boost::apply_visitor(outputHeightVisitor, network.front());
}
}
for (size_t i = 1; i < network.size(); ++i)
{
if (!reset)
{
// Set the input width.
boost::apply_visitor(SetInputWidthVisitor(width), network[i]);
// Set the input height.
boost::apply_visitor(SetInputHeightVisitor(height), network[i]);
}
boost::apply_visitor(ForwardVisitor(boost::apply_visitor(
outputParameterVisitor, network[i - 1]),
boost::apply_visitor(outputParameterVisitor, network[i])),
network[i]);
if (!reset)
{
// Get the output width.
if (boost::apply_visitor(outputWidthVisitor, network[i]) != 0)
{
width = boost::apply_visitor(outputWidthVisitor, network[i]);
}
// Get the output height.
if (boost::apply_visitor(outputHeightVisitor, network[i]) != 0)
{
height = boost::apply_visitor(outputHeightVisitor, network[i]);
}
}
}
if (!reset)
{
reset = true;
}
output = boost::apply_visitor(outputParameterVisitor, network.back());
if (arma::size(output) != arma::size(input))
{
Log::Fatal << "The sizes of the output and input matrices of the Highway"
<< " network should be equal. Please examine the network layers.";
}
transformGate = transformWeight * input;
transformGate.each_col() += transformBias;
transformGateActivation = 1.0 /(1 + arma::exp(-transformGate));
inputParameter = input;
networkOutput = output;
output = (output % transformGateActivation) +
(input % (1 - transformGateActivation));
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void Highway<InputDataType, OutputDataType, CustomLayers...>::Backward(
const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g)
{
arma::Mat<eT> gyTransform = gy % transformGateActivation;
boost::apply_visitor(BackwardVisitor(boost::apply_visitor(
outputParameterVisitor, network.back()),
gyTransform,
boost::apply_visitor(deltaVisitor, network.back())),
network.back());
for (size_t i = 2; i < network.size() + 1; ++i)
{
boost::apply_visitor(BackwardVisitor(boost::apply_visitor(
outputParameterVisitor, network[network.size() - i]),
boost::apply_visitor(deltaVisitor, network[network.size() - i + 1]),
boost::apply_visitor(deltaVisitor,
network[network.size() - i])), network[network.size() - i]);
}
g = boost::apply_visitor(deltaVisitor, network.front());
transformGateError = gy % (networkOutput - inputParameter) %
transformGateActivation % (1.0 - transformGateActivation);
g += transformWeight.t() * transformGateError;
g += gy % (1 - transformGateActivation);
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void Highway<InputDataType, OutputDataType, CustomLayers...>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient)
{
arma::Mat<eT> errorTransform = error % transformGateActivation;
boost::apply_visitor(GradientVisitor(boost::apply_visitor(
outputParameterVisitor, network[network.size() - 2]),
errorTransform), network.back());
for (size_t i = 2; i < network.size(); ++i)
{
boost::apply_visitor(GradientVisitor(boost::apply_visitor(
outputParameterVisitor, network[network.size() - i - 1]),
boost::apply_visitor(deltaVisitor, network[network.size() - i + 1])),
network[network.size() - i]);
}
boost::apply_visitor(GradientVisitor(input,
boost::apply_visitor(deltaVisitor, network[1])), network.front());
gradient.submat(0, 0, transformWeight.n_elem - 1, 0) = arma::vectorise(
transformGateError * input.t());
gradient.submat(transformWeight.n_elem, 0, gradient.n_elem - 1, 0) =
arma::sum(transformGateError, 1);
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename Archive>
void Highway<InputDataType, OutputDataType, CustomLayers...>::serialize(
Archive& ar, const uint32_t /* version */)
{
// If loading, delete the old layers and set size for weights.
if (cereal::is_loading<Archive>())
{
for (LayerTypes<CustomLayers...>& layer : network)
{
boost::apply_visitor(deleteVisitor, layer);
}
weights.set_size(inSize * inSize + inSize, 1);
}
ar(CEREAL_NVP(model));
ar(CEREAL_VECTOR_VARIANT_POINTER(network));
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/layer.hpp
+308 -69
View File
@@ -2,7 +2,7 @@
* @file methods/ann/layer/layer.hpp
* @author Marcus Edel
*
* This includes various layers to construct a model.
* Base class for neural network layers.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
@@ -12,73 +12,312 @@
#ifndef MLPACK_METHODS_ANN_LAYER_LAYER_HPP
#define MLPACK_METHODS_ANN_LAYER_LAYER_HPP
#include "add.hpp"
#include "adaptive_max_pooling.hpp"
#include "adaptive_mean_pooling.hpp"
#include "add_merge.hpp"
#include "alpha_dropout.hpp"
#include "atrous_convolution.hpp"
#include "base_layer.hpp"
#include "batch_norm.hpp"
#include "bicubic_interpolation.hpp"
#include "bilinear_interpolation.hpp"
#include "c_relu.hpp"
#include "celu.hpp"
#include "concat_performance.hpp"
#include "concat.hpp"
#include "concatenate.hpp"
#include "constant.hpp"
#include "convolution.hpp"
#include "dropconnect.hpp"
#include "dropout.hpp"
#include "elu.hpp"
#include "fast_lstm.hpp"
#include "flatten_t_swish.hpp"
#include "flexible_relu.hpp"
#include "glimpse.hpp"
#include "gru.hpp"
#include "hard_tanh.hpp"
#include "hardshrink.hpp"
#include "highway.hpp"
#include "instance_norm.hpp"
#include "join.hpp"
#include "layer_norm.hpp"
#include "layer_types.hpp"
#include "leaky_relu.hpp"
#include "linear.hpp"
#include "linear_no_bias.hpp"
#include "linear3d.hpp"
#include "log_softmax.hpp"
#include "lookup.hpp"
#include "lp_pooling.hpp"
#include "lstm.hpp"
#include "max_pooling.hpp"
#include "mean_pooling.hpp"
#include "minibatch_discrimination.hpp"
#include "multihead_attention.hpp"
#include "multiply_constant.hpp"
#include "multiply_merge.hpp"
#include "nearest_interpolation.hpp"
#include "noisylinear.hpp"
#include "padding.hpp"
#include "parametric_relu.hpp"
#include "pixel_shuffle.hpp"
#include "positional_encoding.hpp"
#include "recurrent_attention.hpp"
#include "recurrent.hpp"
#include "reinforce_normal.hpp"
#include "relu6.hpp"
#include "reparametrization.hpp"
#include "select.hpp"
#include "sequential.hpp"
#include "softshrink.hpp"
#include "softmax.hpp"
#include "softmin.hpp"
#include "spatial_dropout.hpp"
#include "subview.hpp"
#include "transposed_convolution.hpp"
#include "virtual_batch_norm.hpp"
#include "vr_class_reward.hpp"
#include "weight_norm.hpp"
namespace mlpack {
namespace ann {
/**
* A layer is an abstract class implementing common neural networks operations,
* such as convolution, batch norm, etc. These operations require managing
* weights, losses, updates, and inter-layer connectivity.
*
* Users will just instantiate a layer by inherited from the abstract class and
* implement the layer specific methods. It is recommend that descendants of
* Layer implement the following methods:
*
* - Constructor: Defines custom layer attributes, and creates layer state
* variables.
*
* - Forward(input, output): Performs the forward logic of applying the layer
* to the input object and storing the result in the output object.
*
* - Backward(input, gy, g): Performs a backpropagation step through the layer,
* with respect to the given input.
*
* - Gradient(input, error, gradient): Computing the gradient of the layer with
* respect to its own input.
*
* The memory for the layer's parameters (weights and biases) is not allocated
* by the layer itself, instead it is allocated by the network that the layer
* belongs to, and passed to the layer when it needs to use it.
*
* See the linear layer implementation for a basic example. It's a layer with
* two variables, w and b, that returns y = w * x + b. It shows how to implement
* Forward(), Backward() and Gradient(). The weights of the layers are tracked
* in layer.Parameters().
*
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template<typename MatType = arma::mat>
class Layer
{
public:
//! Default constructor.
Layer() : validOutputDimensions(false), training(false)
{ /* Nothing to do here */ }
//! Default deconstructor.
virtual ~Layer() { /* Nothing to do here */ }
//! Copy constructor. This is not responsible for copying weights!
Layer(const Layer& layer) :
inputDimensions(layer.inputDimensions),
outputDimensions(layer.outputDimensions),
validOutputDimensions(layer.validOutputDimensions),
training(layer.training)
{ /* Nothing to do here */ }
//! Make a copy of the object.
virtual Layer* Clone() const = 0;
//! Move constructor. This is not responsible for moving weights!
Layer(Layer&& layer) :
inputDimensions(std::move(layer.inputDimensions)),
outputDimensions(std::move(layer.outputDimensions)),
validOutputDimensions(std::move(layer.validOutputDimensions)),
training(std::move(layer.training))
{ /* Nothing to do here */ }
//! Copy assignment operator. This is not responsible for copying weights!
virtual Layer& operator=(const Layer& layer)
{
if (&layer != this)
{
inputDimensions = layer.inputDimensions;
outputDimensions = layer.outputDimensions;
validOutputDimensions = layer.validOutputDimensions;
training = layer.training;
}
return *this;
}
//! Move assignment operator. This is not responsible for moving weights!
virtual Layer& operator=(Layer&& layer)
{
if (&layer != this)
{
inputDimensions = std::move(layer.inputDimensions);
outputDimensions = std::move(layer.outputDimensions);
validOutputDimensions = std::move(layer.validOutputDimensions);
training = std::move(layer.training);
}
return *this;
}
/**
* Takes an input object, and computes the corresponding output of the layer.
* In general input and output are matrices. However, some special layers like
* table layers might expect something else. Please, refer to each layer
* specification for further information.
*
* @param * (input) Input data used for evaluating the specified layer.
* @param * (output) Resulting output.
*/
virtual void Forward(const MatType& /* input */,
MatType& /* output */)
{ /* Nothing to do here */ }
/**
* Takes an input and output object, and computes the corresponding loss of
* the layer. In general input and output are matrices. However, some special
* layers like table layers might expect something else. Please, refer to each
* layer specification for further information.
*
* @param * (input) Input data used for evaluating the specified layer.
* @param * (output) Resulting output.
*/
virtual void Forward(const MatType& /* input */,
const MatType& /* output */)
{ /* Nothing to do here */ }
/**
* Performs a backpropagation step through the layer, with respect to the
* given input. In general this method makes the assumption Forward(input,
* output) has been called before, with the same input. If you do not respect
* this rule, Backward(input, gy, g) might compute incorrect results.
*
* In general input and gy and g are matrices. However, some special
* sub-classes like table layers might expect something else. Please, refer to
* each module specification for further information.
*
* A backpropagation step consist of computing of computing the gradient
* output input with respect to the output of the layer and given error.
*
* During the backward pass our goal is to use 'gy' in order to compute the
* downstream gradients (g). We assume that the upstream gradient (gy) has
* already been computed and is passed to the layer.
*
* @param * (input) The propagated input activation.
* @param * (gy) The backpropagated error.
* @param * (g) The calculated gradient.
*/
virtual void Backward(const MatType& /* input */,
const MatType& /* gy */,
MatType& /* g */)
{ /* Nothing to do here */ }
/**
* Computing the gradient of the layer with respect to its own input. This is
* returned in gradient.
*
* The layer parameters (weights and biases) are updated accordingly using the
* computed gradient not by the layer itself, instead they are updated by the
* network that holds the instantiated layer.
*
* @param * (input) The input parameter used for calculating the gradient.
* @param * (error) The calculated error.
* @param * (gradient) The calculated gradient.
*/
virtual void Gradient(const MatType& /* input */,
const MatType& /* error */,
MatType& /* gradient */)
{ /* Nothing to do here */ }
/**
* Reset the layer parameter. The method is called to assigned the allocated
* memory to the internal layer parameters like weights and biases. The method
* should be called before the first call of Forward(input, output). If you
* do not respect this rule, Forward(input, output) and Backward(input, gy, g)
* might compute incorrect results.
*
* @param weightsPtr This pointer should be used as the first element of the
* memory that is allocated for this layer. In general, SetWeights()
* implementations should use MakeAlias() with weightsPtr to wrap the
* weights of a layer.
*/
virtual void SetWeights(typename MatType::elem_type* /* weightsPtr */) { }
/**
* Get the total number of trainable weights in the layer.
*/
virtual size_t WeightSize() const { return 0; }
/**
* Get whether the layer is currently in training mode.
*
* @note During network training, this should be set to `true` for each layer
* in the network, and when predicting/testing the network, this should be set
* to `false`. (This is handled automatically by the `FFN` class and other
* related classes.)
*/
virtual bool const& Training() const { return training; }
/**
* Modify whether the layer is currently in training mode.
*
* @note During network training, this should be set to `true` for each layer
* in the network, and when predicting/testing the network, this should be set
* to `false`. (This is handled automatically by the `FFN` class and other
* related classes.)
*/
virtual bool& Training() { return training; }
//! Get the layer loss. Overload this if the layer should add any extra loss
//! to the loss function when computing the objective. (TODO: better comment)
virtual double Loss() { return 0; }
//! Get the input dimensions.
const std::vector<size_t>& InputDimensions() const { return inputDimensions; }
//! Modify the input dimensions.
std::vector<size_t>& InputDimensions()
{
validOutputDimensions = false;
return inputDimensions;
}
//! Get the output dimensions.
const std::vector<size_t>& OutputDimensions()
{
if (!validOutputDimensions)
{
this->ComputeOutputDimensions();
validOutputDimensions = true;
}
return outputDimensions;
}
//! Get the parameters.
virtual const MatType& Parameters() const
{
throw std::invalid_argument("Layer::Parameters(): cannot access parameters "
"of a layer with no weights!");
}
//! Set the parameters.
virtual MatType& Parameters()
{
throw std::invalid_argument("Layer::Parameters(): cannot modify parameters "
"of a layer with no weights!");
}
//! Compute the output dimensions. This should be overloaded if the layer is
//! meant to work on higher-dimensional objects. When this is called, it is a
//! safe assumption that InputDimensions() is correct.
virtual void ComputeOutputDimensions()
{
// The default implementation is to assume that the output size is the same
// as the input.
outputDimensions = inputDimensions;
}
//! Get the number of elements in the output from this layer. This cannot be
//! overloaded! Overload `ComputeOutputDimensions()` instead.
virtual size_t OutputSize() final
{
if (!validOutputDimensions)
{
this->ComputeOutputDimensions();
validOutputDimensions = true;
}
size_t outputSize = 1;
for (size_t i = 0; i < this->outputDimensions.size(); ++i)
outputSize *= this->outputDimensions[i];
return outputSize;
}
//! Serialize the layer.
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */)
{
ar(CEREAL_NVP(inputDimensions));
ar(CEREAL_NVP(outputDimensions));
ar(CEREAL_NVP(validOutputDimensions));
ar(CEREAL_NVP(training));
// Note that layer weights are serialized by the FFN!
}
protected:
/**
* Logical input dimensions of each point. Although each point given to !
* `Forward()` will be represented as a column in a matrix, logically
* speaking it can be a higher-order tensor. So, for instance, if the point
* is 2-dimensional images of size 10x10, `Forward()` will contain columns
* with 100 rows, and `inputDimensions` will be `{10, 10}`. This generalizes
* to higher dimensions.
*/
std::vector<size_t> inputDimensions;
/**
* Logical output dimensions of each point. If the layer only performs
* elementwise operations, this is most likely equal to `inputDimensions`; but
* if the layer performs more complicated transformations, it may be
* different.
*/
std::vector<size_t> outputDimensions;
//! This is `true` if `ComputeOutputDimensions()` has been called, and
//! `outputDimensions` can be considered to be up-to-date.
bool validOutputDimensions;
//! If true, the layer is in training mode; otherwise, it is in testing mode.
bool training;
};
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/layer_traits.hpp
-130
View File
@@ -1,130 +0,0 @@
/**
* @file methods/ann/layer/layer_traits.hpp
* @author Marcus Edel
*
* This provides the LayerTraits class, a template class to get information
* about various layers.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_LAYER_TRAITS_HPP
#define MLPACK_METHODS_ANN_LAYER_LAYER_TRAITS_HPP
#include <mlpack/core/util/sfinae_utility.hpp>
namespace mlpack {
namespace ann {
/**
* This is a template class that can provide information about various layers.
* By default, this class will provide the weakest possible assumptions on
* layer, and each layer should override values as necessary. If a layer
* doesn't need to override a value, then there's no need to write a LayerTraits
* specialization for that class.
*/
template<typename LayerType>
class LayerTraits
{
public:
/**
* This is true if the layer is a binary layer.
*/
static const bool IsBinary = false;
/**
* This is true if the layer is an output layer.
*/
static const bool IsOutputLayer = false;
/**
* This is true if the layer is a bias layer.
*/
static const bool IsBiasLayer = false;
/*
* This is true if the layer is a LSTM layer.
**/
static const bool IsLSTMLayer = false;
/*
* This is true if the layer is a connection layer.
**/
static const bool IsConnection = false;
};
// This gives us a HasGradientCheck<T, U> type (where U is a function pointer)
// we can use with SFINAE to catch when a type has a Gradient(...) function.
HAS_MEM_FUNC(Gradient, HasGradientCheck);
// This gives us a HasDeterministicCheck<T, U> type (where U is a function
// pointer) we can use with SFINAE to catch when a type has a Deterministic()
// function.
HAS_MEM_FUNC(Deterministic, HasDeterministicCheck);
// This gives us a HasParametersCheck<T, U> type (where U is a function pointer)
// we can use with SFINAE to catch when a type has a Parameters() function.
HAS_MEM_FUNC(Parameters, HasParametersCheck);
// This gives us a HasAddCheck<T, U> type (where U is a function pointer) we
// can use with SFINAE to catch when a type has a Add() function.
HAS_MEM_FUNC(Add, HasAddCheck);
// This gives us a HasModelCheck<T> type we can use with SFINAE to catch when
// a type has a function named Model.
HAS_ANY_METHOD_FORM(Model, HasModelCheck);
// This gives us a HasLocationCheck<T, U> type (where U is a function pointer)
// we can use with SFINAE to catch when a type has a Location() function.
HAS_MEM_FUNC(Location, HasLocationCheck);
// This gives us a HasResetCheck<T, U> type (where U is a function pointer)
// we can use with SFINAE to catch when a type has a Reset() function.
HAS_MEM_FUNC(Reset, HasResetCheck);
// This gives us a HasResetCheck<T, U> type (where U is a function pointer)
// we can use with SFINAE to catch when a type has a ResetCell() function.
HAS_MEM_FUNC(ResetCell, HasResetCellCheck);
// This gives us a HasRewardCheck<T, U> type (where U is a function pointer) we
// can use with SFINAE to catch when a type has a Reward() function.
HAS_MEM_FUNC(Reward, HasRewardCheck);
// This gives us a HasInputWidth<T, U> type (where U is a function pointer) we
// can use with SFINAE to catch when a type has a InputWidth() function.
HAS_MEM_FUNC(InputWidth, HasInputWidth);
// This gives us a HasInputHeight<T, U> type (where U is a function pointer) we
// can use with SFINAE to catch when a type has a InputHeight() function.
HAS_MEM_FUNC(InputHeight, HasInputHeight);
// This gives us a HasRho<T, U> type (where U is a function pointer) we
// can use with SFINAE to catch when a type has a Rho() function.
HAS_MEM_FUNC(Rho, HasRho);
// This gives us a HasLoss<T, U> type (where U is a function pointer) we
// can use with SFINAE to catch when a type has a Loss() function.
HAS_MEM_FUNC(Loss, HasLoss);
// This gives us a HasRunCheck<T, U> type (where U is a function pointer) we
// can use with SFINAE to catch when a type has a Run() function.
HAS_MEM_FUNC(Run, HasRunCheck);
// This gives us a HasBiasCheck<T, U> type (where U is a function pointer) we
// can use with SFINAE to catch when a type has a Bias() function.
HAS_MEM_FUNC(Bias, HasBiasCheck);
// This gives us a HasMaxIterationsC<T, U> type (where U is a function pointer)
// we can use with SFINAE to catch when a type has a MaxIterations() function.
HAS_MEM_FUNC(MaxIterations, HasMaxIterations);
// This gives us a HasInShapeCheck<T> type we can use with SFINAE to catch when
// a type has a function named InputShape.
HAS_ANY_METHOD_FORM(InputShape, HasInputShapeCheck);
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/layer_types.hpp
+13 -285
View File
@@ -12,62 +12,33 @@
#ifndef MLPACK_METHODS_ANN_LAYER_LAYER_TYPES_HPP
#define MLPACK_METHODS_ANN_LAYER_LAYER_TYPES_HPP
#include <boost/variant.hpp>
#include <mlpack/methods/ann/layer/layer.hpp>
#include <mlpack/methods/ann/layer/multi_layer.hpp>
#include <mlpack/methods/ann/layer/recurrent_layer.hpp>
// Layer modules.
// Include each layer.
#include <mlpack/methods/ann/layer/add.hpp>
#include <mlpack/methods/ann/layer/alpha_dropout.hpp>
#include <mlpack/methods/ann/layer/base_layer.hpp>
#include <mlpack/methods/ann/layer/batch_norm.hpp>
#include <mlpack/methods/ann/layer/bicubic_interpolation.hpp>
#include <mlpack/methods/ann/layer/bilinear_interpolation.hpp>
#include <mlpack/methods/ann/layer/channel_shuffle.hpp>
#include <mlpack/methods/ann/layer/constant.hpp>
#include <mlpack/methods/ann/layer/concatenate.hpp>
#include <mlpack/methods/ann/layer/convolution.hpp>
#include <mlpack/methods/ann/layer/dropconnect.hpp>
#include <mlpack/methods/ann/layer/dropout.hpp>
#include <mlpack/methods/ann/layer/elu.hpp>
#include <mlpack/methods/ann/layer/hard_tanh.hpp>
#include <mlpack/methods/ann/layer/instance_norm.hpp>
#include <mlpack/methods/ann/layer/group_norm.hpp>
#include <mlpack/methods/ann/layer/join.hpp>
#include <mlpack/methods/ann/layer/layer_norm.hpp>
#include <mlpack/methods/ann/layer/leaky_relu.hpp>
#include <mlpack/methods/ann/layer/c_relu.hpp>
#include <mlpack/methods/ann/layer/flexible_relu.hpp>
#include <mlpack/methods/ann/layer/linear.hpp>
#include <mlpack/methods/ann/layer/linear_no_bias.hpp>
#include <mlpack/methods/ann/layer/linear3d.hpp>
#include <mlpack/methods/ann/layer/log_softmax.hpp>
#include <mlpack/methods/ann/layer/lookup.hpp>
#include <mlpack/methods/ann/layer/multihead_attention.hpp>
#include <mlpack/methods/ann/layer/multiply_constant.hpp>
#include <mlpack/methods/ann/layer/lstm.hpp>
#include <mlpack/methods/ann/layer/max_pooling.hpp>
#include <mlpack/methods/ann/layer/mean_pooling.hpp>
#include <mlpack/methods/ann/layer/lp_pooling.hpp>
#include <mlpack/methods/ann/layer/nearest_interpolation.hpp>
#include <mlpack/methods/ann/layer/noisylinear.hpp>
#include <mlpack/methods/ann/layer/adaptive_max_pooling.hpp>
#include <mlpack/methods/ann/layer/adaptive_mean_pooling.hpp>
#include <mlpack/methods/ann/layer/parametric_relu.hpp>
#include <mlpack/methods/ann/layer/pixel_shuffle.hpp>
#include <mlpack/methods/ann/layer/positional_encoding.hpp>
#include <mlpack/methods/ann/layer/reinforce_normal.hpp>
#include <mlpack/methods/ann/layer/relu6.hpp>
#include <mlpack/methods/ann/layer/reparametrization.hpp>
#include <mlpack/methods/ann/layer/select.hpp>
#include <mlpack/methods/ann/layer/softmax.hpp>
#include <mlpack/methods/ann/layer/spatial_dropout.hpp>
#include <mlpack/methods/ann/layer/subview.hpp>
#include <mlpack/methods/ann/layer/virtual_batch_norm.hpp>
#include <mlpack/methods/ann/layer/hardshrink.hpp>
#include <mlpack/methods/ann/layer/celu.hpp>
#include <mlpack/methods/ann/layer/isrlu.hpp>
#include <mlpack/methods/ann/layer/softshrink.hpp>
#include <mlpack/methods/ann/layer/padding.hpp>
#include <mlpack/methods/ann/layer/radial_basis_function.hpp>
// Convolution modules.
// Convolution modes.
#include <mlpack/methods/ann/convolution_rules/border_modes.hpp>
#include <mlpack/methods/ann/convolution_rules/naive_convolution.hpp>
#include <mlpack/methods/ann/convolution_rules/fft_convolution.hpp>
#include <mlpack/methods/ann/convolution_rules/naive_convolution.hpp>
// Regularizers.
#include <mlpack/methods/ann/regularizer/no_regularizer.hpp>
@@ -75,250 +46,7 @@
// Loss function modules.
#include <mlpack/methods/ann/loss_functions/negative_log_likelihood.hpp>
namespace mlpack {
namespace ann {
template<typename InputDataType, typename OutputDataType> class BatchNorm;
template<typename InputDataType, typename OutputDataType> class DropConnect;
template<typename InputDataType, typename OutputDataType> class Glimpse;
template<typename InputDataType, typename OutputDataType> class LayerNorm;
template<typename InputDataType, typename OutputDataType> class LSTM;
template<typename InputDataType, typename OutputDataType> class GRU;
template<typename InputDataType, typename OutputDataType> class FastLSTM;
template<typename InputDataType, typename OutputDataType> class VRClassReward;
template<typename InputDataType, typename OutputDataType> class Concatenate;
template<typename InputDataType, typename OutputDataType> class Padding;
template<typename InputDataType, typename OutputDataType> class ReLU6;
template<typename InputDataType,
typename OutputDataType,
typename RegularizerType>
class Linear;
template<typename InputDataType,
typename OutputDataType,
typename Activation>
class RBF;
template<typename InputDataType,
typename OutputDataType,
typename RegularizerType>
class LinearNoBias;
template<typename InputDataType,
typename OutputDataType>
class NoisyLinear;
template<typename InputDataType,
typename OutputDataType,
typename RegularizerType>
class Linear3D;
template<typename InputDataType,
typename OutputDataType
>
class VirtualBatchNorm;
template<typename InputDataType,
typename OutputDataType
>
class MiniBatchDiscrimination;
template <typename InputDataType,
typename OutputDataType,
typename RegularizerType>
class MultiheadAttention;
template<typename InputDataType,
typename OutputDataType
>
class Reparametrization;
template<typename InputDataType,
typename OutputDataType,
typename... CustomLayers
>
class AddMerge;
template<typename InputDataType,
typename OutputDataType,
bool residual,
typename... CustomLayers
>
class Sequential;
template<typename InputDataType,
typename OutputDataType,
typename... CustomLayers
>
class Highway;
template<typename InputDataType,
typename OutputDataType,
typename... CustomLayers
>
class Recurrent;
template<typename InputDataType,
typename OutputDataType,
typename... CustomLayers
>
class Concat;
template<
typename OutputLayerType,
typename InputDataType,
typename OutputDataType
>
class ConcatPerformance;
template<
typename ForwardConvolutionRule,
typename BackwardConvolutionRule,
typename GradientConvolutionRule,
typename InputDataType,
typename OutputDataType
>
class Convolution;
template<
typename ForwardConvolutionRule,
typename BackwardConvolutionRule,
typename GradientConvolutionRule,
typename InputDataType,
typename OutputDataType
>
class TransposedConvolution;
template<
typename ForwardConvolutionRule,
typename BackwardConvolutionRule,
typename GradientConvolutionRule,
typename InputDataType,
typename OutputDataType
>
class AtrousConvolution;
template<
typename InputDataType,
typename OutputDataType
>
class RecurrentAttention;
template<typename InputDataType,
typename OutputDataType,
typename... CustomLayers
>
class MultiplyMerge;
template <typename InputDataType,
typename OutputDataType,
typename... CustomLayers
>
class WeightNorm;
template <typename InputDataType,
typename OutputDataType
>
class AdaptiveMaxPooling;
template <typename InputDataType,
typename OutputDataType
>
class AdaptiveMeanPooling;
using MoreTypes = boost::variant<
FlexibleReLU<arma::mat, arma::mat>*,
Linear3D<arma::mat, arma::mat, NoRegularizer>*,
LpPooling<arma::mat, arma::mat>*,
PixelShuffle<arma::mat, arma::mat>*,
ChannelShuffle<arma::mat, arma::mat>*,
Glimpse<arma::mat, arma::mat>*,
Highway<arma::mat, arma::mat>*,
MultiheadAttention<arma::mat, arma::mat, NoRegularizer>*,
Recurrent<arma::mat, arma::mat>*,
RecurrentAttention<arma::mat, arma::mat>*,
ReinforceNormal<arma::mat, arma::mat>*,
ReLU6<arma::mat, arma::mat>*,
Reparametrization<arma::mat, arma::mat>*,
Select<arma::mat, arma::mat>*,
SpatialDropout<arma::mat, arma::mat>*,
Subview<arma::mat, arma::mat>*,
VRClassReward<arma::mat, arma::mat>*,
VirtualBatchNorm<arma::mat, arma::mat>*,
RBF<arma::mat, arma::mat, GaussianFunction>*,
BaseLayer<GaussianFunction, arma::mat, arma::mat>*,
PositionalEncoding<arma::mat, arma::mat>*,
ISRLU<arma::mat, arma::mat>*,
BicubicInterpolation<arma::mat, arma::mat>*,
NearestInterpolation<arma::mat, arma::mat>*,
GroupNorm<arma::mat, arma::mat>*,
InstanceNorm<arma::mat, arma::mat>*
>;
template <typename... CustomLayers>
using LayerTypes = boost::variant<
AdaptiveMaxPooling<arma::mat, arma::mat>*,
AdaptiveMeanPooling<arma::mat, arma::mat>*,
Add<arma::mat, arma::mat>*,
AddMerge<arma::mat, arma::mat>*,
AlphaDropout<arma::mat, arma::mat>*,
AtrousConvolution<NaiveConvolution<ValidConvolution>,
NaiveConvolution<FullConvolution>,
NaiveConvolution<ValidConvolution>,
arma::mat, arma::mat>*,
BaseLayer<LogisticFunction, arma::mat, arma::mat>*,
BaseLayer<IdentityFunction, arma::mat, arma::mat>*,
BaseLayer<TanhFunction, arma::mat, arma::mat>*,
BaseLayer<SoftplusFunction, arma::mat, arma::mat>*,
BaseLayer<RectifierFunction, arma::mat, arma::mat>*,
BatchNorm<arma::mat, arma::mat>*,
BilinearInterpolation<arma::mat, arma::mat>*,
CELU<arma::mat, arma::mat>*,
Concat<arma::mat, arma::mat>*,
Concatenate<arma::mat, arma::mat>*,
ConcatPerformance<NegativeLogLikelihood<arma::mat, arma::mat>,
arma::mat, arma::mat>*,
Constant<arma::mat, arma::mat>*,
Convolution<NaiveConvolution<ValidConvolution>,
NaiveConvolution<FullConvolution>,
NaiveConvolution<ValidConvolution>, arma::mat, arma::mat>*,
CReLU<arma::mat, arma::mat>*,
DropConnect<arma::mat, arma::mat>*,
Dropout<arma::mat, arma::mat>*,
ELU<arma::mat, arma::mat>*,
FastLSTM<arma::mat, arma::mat>*,
GRU<arma::mat, arma::mat>*,
HardTanH<arma::mat, arma::mat>*,
Join<arma::mat, arma::mat>*,
LayerNorm<arma::mat, arma::mat>*,
LeakyReLU<arma::mat, arma::mat>*,
Linear<arma::mat, arma::mat, NoRegularizer>*,
LinearNoBias<arma::mat, arma::mat, NoRegularizer>*,
LogSoftMax<arma::mat, arma::mat>*,
Lookup<arma::mat, arma::mat>*,
LSTM<arma::mat, arma::mat>*,
MaxPooling<arma::mat, arma::mat>*,
MeanPooling<arma::mat, arma::mat>*,
MiniBatchDiscrimination<arma::mat, arma::mat>*,
MultiplyConstant<arma::mat, arma::mat>*,
MultiplyMerge<arma::mat, arma::mat>*,
NegativeLogLikelihood<arma::mat, arma::mat>*,
NoisyLinear<arma::mat, arma::mat>*,
Padding<arma::mat, arma::mat>*,
PReLU<arma::mat, arma::mat>*,
Sequential<arma::mat, arma::mat, false>*,
Sequential<arma::mat, arma::mat, true>*,
Softmax<arma::mat, arma::mat>*,
TransposedConvolution<NaiveConvolution<ValidConvolution>,
NaiveConvolution<ValidConvolution>,
NaiveConvolution<ValidConvolution>, arma::mat, arma::mat>*,
WeightNorm<arma::mat, arma::mat>*,
MoreTypes,
CustomLayers*...
>;
} // namespace ann
} // namespace mlpack
// Include definitions for polymorphic serialization.
#include <mlpack/methods/ann/layer/serialization.hpp>
#endif
src/mlpack/methods/ann/layer/leaky_relu.hpp
+33 -38
View File
@@ -16,6 +16,8 @@
#include <mlpack/prereqs.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -32,16 +34,11 @@ namespace ann /** Artificial Neural Network. */ {
* \right.
* @f}
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class LeakyReLU
template<typename MatType = arma::mat>
class LeakyReLUType : public Layer<MatType>
{
public:
/**
@@ -49,9 +46,24 @@ class LeakyReLU
* The non zero gradient can be adjusted by specifying the parameter
* alpha in the range 0 to 1. Default (alpha = 0.03)
*
* @param alpha Non zero gradient
* @param alpha Non zero gradient.
*/
LeakyReLU(const double alpha = 0.03);
LeakyReLUType(const double alpha = 0.03);
//! Clone the LeakyReLUType object. This handles polymorphism correctly.
LeakyReLUType* Clone() const { return new LeakyReLUType(*this); }
// Virtual destructor.
virtual ~LeakyReLUType() { }
//! Copy the given LeakyReLUType.
LeakyReLUType(const LeakyReLUType& other);
//! Take ownership of the given LeakyReLUType.
LeakyReLUType(LeakyReLUType&& other);
//! Copy the given LeakyReLUType.
LeakyReLUType& operator=(const LeakyReLUType& other);
//! Take ownership of the given LeakyReLUType.
LeakyReLUType& operator=(LeakyReLUType&& other);
/**
* Ordinary feed forward pass of a neural network, evaluating the function
@@ -60,8 +72,7 @@ class LeakyReLU
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename InputType, typename OutputType>
void Forward(const InputType& input, OutputType& output);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of a neural network, calculating the function
@@ -72,43 +83,27 @@ class LeakyReLU
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename DataType>
void Backward(const DataType& input, const DataType& gy, DataType& g);
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
void Backward(const MatType& input, const MatType& gy, MatType& g);
//! Get the non zero gradient.
double const& Alpha() const { return alpha; }
//! Modify the non zero gradient.
double& Alpha() { return alpha; }
//! Get size of weights.
size_t WeightSize() const { return 0; }
/**
* Serialize the layer.
*/
//! Serialize the layer.
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
//! Leakyness Parameter in the range 0 <alpha< 1
double alpha;
}; // class LeakyReLU
}; // class LeakyReLUType
// Convenience typedefs.
// Standard LeakyReLU layer.
typedef LeakyReLUType<arma::mat> LeakyReLU;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/leaky_relu_impl.hpp
+57 -14
View File
@@ -20,27 +20,68 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
LeakyReLU<InputDataType, OutputDataType>::LeakyReLU(
const double alpha) : alpha(alpha)
template<typename MatType>
LeakyReLUType<MatType>::LeakyReLUType(const double alpha) :
Layer<MatType>(),
alpha(alpha)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
void LeakyReLU<InputDataType, OutputDataType>::Forward(
const InputType& input, OutputType& output)
template<typename MatType>
LeakyReLUType<MatType>::LeakyReLUType(const LeakyReLUType& other) :
Layer<MatType>(other),
alpha(other.alpha)
{
// Nothing to do.
}
template<typename MatType>
LeakyReLUType<MatType>::LeakyReLUType(
LeakyReLUType&& other) :
Layer<MatType>(std::move(other)),
alpha(std::move(other.alpha))
{
// Nothing to do.
}
template<typename MatType>
LeakyReLUType<MatType>&
LeakyReLUType<MatType>::operator=(const LeakyReLUType& other)
{
if (&other != this)
{
Layer<MatType>::operator=(other);
alpha = other.alpha;
}
return *this;
}
template<typename MatType>
LeakyReLUType<MatType>&
LeakyReLUType<MatType>::operator=(LeakyReLUType&& other)
{
if (&other != this)
{
Layer<MatType>::operator=(std::move(other));
alpha = std::move(other.alpha);
}
return *this;
}
template<typename MatType>
void LeakyReLUType<MatType>::Forward(const MatType& input, MatType& output)
{
output = arma::max(input, alpha * input);
}
template<typename InputDataType, typename OutputDataType>
template<typename DataType>
void LeakyReLU<InputDataType, OutputDataType>::Backward(
const DataType& input, const DataType& gy, DataType& g)
template<typename MatType>
void LeakyReLUType<MatType>::Backward(
const MatType& input, const MatType& gy, MatType& g)
{
DataType derivative;
MatType derivative;
derivative.set_size(arma::size(input));
for (size_t i = 0; i < input.n_elem; ++i)
derivative(i) = (input(i) >= 0) ? 1 : alpha;
@@ -48,12 +89,14 @@ void LeakyReLU<InputDataType, OutputDataType>::Backward(
g = gy % derivative;
}
template<typename InputDataType, typename OutputDataType>
template<typename MatType>
template<typename Archive>
void LeakyReLU<InputDataType, OutputDataType>::serialize(
void LeakyReLUType<MatType>::serialize(
Archive& ar,
const uint32_t /* version */)
{
ar(cereal::base_class<Layer<MatType>>(this));
ar(CEREAL_NVP(alpha));
}
src/mlpack/methods/ann/layer/linear.hpp
+72 -100
View File
@@ -16,7 +16,7 @@
#include <mlpack/prereqs.hpp>
#include <mlpack/methods/ann/regularizer/no_regularizer.hpp>
#include "layer_types.hpp"
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -25,142 +25,120 @@ namespace ann /** Artificial Neural Network. */ {
* Implementation of the Linear layer class. The Linear class represents a
* single layer of a neural network.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* The linear layer applies a linear transformation to the incoming data
* (input), i.e. y = Ax + b. The input matrix given in Forward(input, output)
* must be either a vector or matrix. If the input is a matrix, then each column
* is assumed to be an input sample of given batch.
*
* @tparam MatType Matrix representation to accept as input and use for
* computation.
* @tparam RegularizerType Type of the regularizer to be used (Default no
* regularizer).
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat,
template<
typename MatType = arma::mat,
typename RegularizerType = NoRegularizer
>
class Linear
class LinearType : public Layer<MatType>
{
public:
//! Create the Linear object.
Linear();
LinearType();
/**
* Create the Linear layer object using the specified number of units.
* Create the Linear layer object with the specified number of output
* dimensions.
*
* @param inSize The number of input units.
* @param outSize The number of output units.
* @param regularizer The regularizer to use, optional.
* @param outSize The output dimension.
* @param regularizer The regularizer to use, optional (default: no
* regularizer).
*/
Linear(const size_t inSize,
const size_t outSize,
RegularizerType regularizer = RegularizerType());
LinearType(const size_t outSize,
RegularizerType regularizer = RegularizerType());
//! Copy constructor.
Linear(const Linear& layer);
virtual ~LinearType() { }
//! Move constructor.
Linear(Linear&&);
//! Clone the LinearType object. This handles polymorphism correctly.
LinearType* Clone() const { return new LinearType(*this); }
//! Copy assignment operator.
Linear& operator=(const Linear& layer);
//! Copy the other Linear layer (but not weights).
LinearType(const LinearType& layer);
//! Move assignment operator.
Linear& operator=(Linear&& layer);
//! Take ownership of the members of the other Linear layer (but not weights).
LinearType(LinearType&& layer);
/*
* Reset the layer parameter.
//! Copy the other Linear layer (but not weights).
LinearType& operator=(const LinearType& layer);
//! Take ownership of the members of the other Linear layer (but not weights).
LinearType& operator=(LinearType&& layer);
/**
* Reset the layer parameter (weights and bias). The method is called to
* assign the allocated memory to the internal learnable parameters.
*/
void Reset();
void SetWeights(typename MatType::elem_type* weightsPtr);
/**
* Ordinary feed forward pass of a neural network, evaluating the function
* f(x) by propagating the activity forward through f.
*
* f(x) is a linear transformation: Ax + b, where x is the given input, x are
* the layer weights and b is the layer bias.
*
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of a neural network, calculating the function
* f(x) by propagating x backwards trough f. Using the results from the feed
* forward pass.
*
* To compute the downstream gradient (g) the chain rule is used.
*
* @param * (input) The propagated input activation.
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
void Backward(const MatType& /* input */,
const MatType& gy,
MatType& g);
/*
/**
* Calculate the gradient using the output delta and the input activation.
*
* @param input The input parameter used for calculating the gradient.
* @param error The calculated error.
* @param gradient The calculated gradient.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient);
void Gradient(const MatType& input,
const MatType& error,
MatType& gradient);
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
const MatType& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
//! Get the input parameter.
InputDataType const& InputParameter() const { return inputParameter; }
//! Modify the input parameter.
InputDataType& InputParameter() { return inputParameter; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the input size.
size_t InputSize() const { return inSize; }
//! Get the output size.
size_t OutputSize() const { return outSize; }
//! Get the gradient.
OutputDataType const& Gradient() const { return gradient; }
//! Modify the gradient.
OutputDataType& Gradient() { return gradient; }
MatType& Parameters() { return weights; }
//! Get the weight of the layer.
OutputDataType const& Weight() const { return weight; }
MatType const& Weight() const { return weight; }
//! Modify the weight of the layer.
OutputDataType& Weight() { return weight; }
MatType& Weight() { return weight; }
//! Get the bias of the layer.
OutputDataType const& Bias() const { return bias; }
MatType const& Bias() const { return bias; }
//! Modify the bias weights of the layer.
OutputDataType& Bias() { return bias; }
MatType& Bias() { return bias; }
//! Get the size of the weights.
size_t WeightSize() const
{
return (inSize * outSize) + outSize;
}
size_t WeightSize() const { return (inSize * outSize) + outSize; }
//! Get the shape of the input.
size_t InputShape() const
{
return inSize;
}
//! Compute the output dimensions of the layer given `InputDimensions()`.
void ComputeOutputDimensions();
/**
* Serialize the layer
*/
//! Serialize the layer.
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
@@ -171,30 +149,24 @@ class Linear
//! Locally-stored number of output units.
size_t outSize;
//! Locally-stored weight object.
OutputDataType weights;
//! Locally-stored weight object. This holds all the weights in a vectorized
//! form; i.e., the weights and the bias.
MatType weights;
//! Locally-stored weight parameters.
OutputDataType weight;
MatType weight;
//! Locally-stored bias term parameters.
OutputDataType bias;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored gradient object.
OutputDataType gradient;
//! Locally-stored input parameter object.
InputDataType inputParameter;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
MatType bias;
//! Locally-stored regularizer object.
RegularizerType regularizer;
}; // class Linear
}; // class LinearType
// Convenience typedefs.
// Standard Linear layer using no regularization.
typedef LinearType<arma::mat, NoRegularizer> Linear;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/linear3d.hpp
+50 -91
View File
@@ -14,9 +14,10 @@
#define MLPACK_METHODS_ANN_LAYER_LINEAR3D_HPP
#include <mlpack/prereqs.hpp>
#include <mlpack/methods/ann/layer/layer_types.hpp>
#include <mlpack/methods/ann/regularizer/no_regularizer.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -27,49 +28,48 @@ namespace ann /** Artificial Neural Network. */ {
* Shape of input : (inSize * nPoints, batchSize)
* Shape of output : (outSize * nPoints, batchSize)
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat,
template<
typename MatType = arma::mat,
typename RegularizerType = NoRegularizer
>
class Linear3D
class Linear3DType : public Layer<MatType>
{
public:
//! Create the Linear3D object.
Linear3D();
Linear3DType();
/**
* Create the Linear3D layer object using the specified number of units.
* Create the Linear3D layer object using the specified number of output
* units.
*
* @param inSize The number of input units.
* @param outSize The number of output units.
* @param regularizer The regularizer to use, optional.
*/
Linear3D(const size_t inSize,
const size_t outSize,
RegularizerType regularizer = RegularizerType());
Linear3DType(const size_t outSize,
RegularizerType regularizer = RegularizerType());
//! Copy constructor.
Linear3D(const Linear3D& layer);
//! Clone the Linear3DType object. This handles polymorphism correctly.
Linear3DType* Clone() const { return new Linear3DType(*this); }
//! Move constructor.
Linear3D(Linear3D&&);
// Virtual destructor.
virtual ~Linear3DType() { }
//! Copy assignment operator.
Linear3D& operator=(const Linear3D& layer);
//! Move assignment operator.
Linear3D& operator=(Linear3D&& layer);
//! Copy the given Linear3DType (but not weights).
Linear3DType(const Linear3DType& other);
//! Take ownership of the given Linear3DType (but not weights).
Linear3DType(Linear3DType&& other);
//! Copy the given Linear3DType (but not weights).
Linear3DType& operator=(const Linear3DType& other);
//! Take ownership of the given Linear3DType (but not weights).
Linear3DType& operator=(Linear3DType&& other);
/*
* Reset the layer parameter.
*/
void Reset();
void SetWeights(typename MatType::elem_type* weightsPtr);
/**
* Ordinary feed forward pass of a neural network, evaluating the function
@@ -78,8 +78,7 @@ class Linear3D
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of a neural network, calculating the function
@@ -90,69 +89,41 @@ class Linear3D
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
void Backward(const MatType& /* input */,
const MatType& gy,
MatType& g);
/*
/**
* Calculate the gradient using the output delta and the input activation.
*
* @param input The input parameter used for calculating the gradient.
* @param error The calculated error.
* @param gradient The calculated gradient.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient);
void Gradient(const MatType& input,
const MatType& error,
MatType& gradient);
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
MatType const& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
//! Get the input parameter.
InputDataType const& InputParameter() const { return inputParameter; }
//! Modify the input parameter.
InputDataType& InputParameter() { return inputParameter; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the input size.
size_t InputSize() const { return inSize; }
//! Get the output size.
size_t OutputSize() const { return outSize; }
//! Get the gradient.
OutputDataType const& Gradient() const { return gradient; }
//! Modify the gradient.
OutputDataType& Gradient() { return gradient; }
MatType& Parameters() { return weights; }
//! Get the weight of the layer.
OutputDataType const& Weight() const { return weight; }
MatType const& Weight() const { return weight; }
//! Modify the weight of the layer.
OutputDataType& Weight() { return weight; }
MatType& Weight() { return weight; }
//! Get the bias of the layer.
OutputDataType const& Bias() const { return bias; }
MatType const& Bias() const { return bias; }
//! Modify the bias weights of the layer.
OutputDataType& Bias() { return bias; }
MatType& Bias() { return bias; }
//! Get the shape of the input.
size_t InputShape() const
{
return inSize;
}
//! Return the number of weight elements.
size_t WeightSize() const { return outSize * (this->inputDimensions[0] + 1); }
//! Compute the output dimensions for the layer, using `InputDimensions()`.
void ComputeOutputDimensions();
/**
* Serialize the layer
@@ -161,37 +132,25 @@ class Linear3D
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Locally-stored number of input units.
size_t inSize;
//! Locally-stored number of output units.
size_t outSize;
//! Locally-stored weight object.
OutputDataType weights;
MatType weights;
//! Locally-stored weight parameters.
OutputDataType weight;
MatType weight;
//! Locally-stored bias term parameters.
OutputDataType bias;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored gradient object.
OutputDataType gradient;
//! Locally-stored input parameter object.
InputDataType inputParameter;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
MatType bias;
//! Locally-stored regularizer object.
RegularizerType regularizer;
}; // class Linear
// Standard Linear3D layer.
typedef Linear3DType<arma::mat, NoRegularizer> Linear3D;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/linear3d_impl.hpp
+87 -111
View File
@@ -18,116 +18,94 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
Linear3D<InputDataType, OutputDataType, RegularizerType>::Linear3D() :
inSize(0),
template<typename MatType, typename RegularizerType>
Linear3DType<MatType, RegularizerType>::Linear3DType() :
Layer<MatType>(),
outSize(0)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
Linear3D<InputDataType, OutputDataType, RegularizerType>::Linear3D(
const size_t inSize,
template<typename MatType, typename RegularizerType>
Linear3DType<MatType, RegularizerType>::Linear3DType(
const size_t outSize,
RegularizerType regularizer) :
inSize(inSize),
Layer<MatType>(),
outSize(outSize),
regularizer(regularizer)
{ }
template<typename MatType, typename RegularizerType>
Linear3DType<MatType, RegularizerType>::Linear3DType(
const Linear3DType& other) :
Layer<MatType>(other),
outSize(other.outSize),
regularizer(other.regularizer)
{
weights.set_size(outSize * inSize + outSize, 1);
// Nothing to do.
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
Linear3D<InputDataType, OutputDataType, RegularizerType>::Linear3D(
const Linear3D& layer) :
inSize(layer.inSize),
outSize(layer.outSize),
weights(layer.weights),
regularizer(layer.regularizer)
template<typename MatType, typename RegularizerType>
Linear3DType<MatType, RegularizerType>::Linear3DType(
Linear3DType&& other) :
Layer<MatType>(std::move(other)),
outSize(std::move(other.outSize)),
regularizer(std::move(other.regularizer))
{
// Nothing to do here.
// Nothing to do.
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
Linear3D<InputDataType, OutputDataType, RegularizerType>::Linear3D(
Linear3D&& layer) :
inSize(0),
outSize(0),
weights(std::move(layer.weights)),
regularizer(std::move(layer.regularizer))
template<typename MatType, typename RegularizerType>
Linear3DType<MatType, RegularizerType>&
Linear3DType<MatType, RegularizerType>::operator=(
const Linear3DType& other)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
Linear3D<InputDataType, OutputDataType, RegularizerType>&
Linear3D<InputDataType, OutputDataType, RegularizerType>::
operator=(const Linear3D& layer)
{
if (this != &layer)
if (&other != this)
{
inSize = layer.inSize;
outSize = layer.outSize;
weights = layer.weights;
regularizer = layer.regularizer;
Layer<MatType>::operator=(other);
outSize = other.outSize;
regularizer = other.regularizer;
}
return *this;
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
Linear3D<InputDataType, OutputDataType, RegularizerType>&
Linear3D<InputDataType, OutputDataType, RegularizerType>::
operator=(Linear3D&& layer)
template<typename MatType, typename RegularizerType>
Linear3DType<MatType, RegularizerType>&
Linear3DType<MatType, RegularizerType>::operator=(
Linear3DType&& other)
{
if (this != &layer)
if (&other != this)
{
inSize = 0;
outSize = 0;
weights = std::move(layer.weights);
regularizer = std::move(layer.regularizer);
Layer<MatType>::operator=(std::move(other));
outSize = std::move(other.outSize);
regularizer = std::move(other.regularizer);
}
return *this;
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
void Linear3D<InputDataType, OutputDataType, RegularizerType>::Reset()
template<typename MatType, typename RegularizerType>
void Linear3DType<MatType, RegularizerType>::SetWeights(
typename MatType::elem_type* weightsPtr)
{
typedef typename arma::Mat<typename OutputDataType::elem_type> MatType;
weight = MatType(weights.memptr(), outSize, inSize, false, false);
bias = MatType(weights.memptr() + weight.n_elem, outSize, 1, false, false);
MakeAlias(weights, weightsPtr, outSize * this->inputDimensions[0] + outSize,
1);
MakeAlias(weight, weightsPtr, outSize, this->inputDimensions[0]);
MakeAlias(bias, weightsPtr + weight.n_elem, outSize, 1);
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
template<typename eT>
void Linear3D<InputDataType, OutputDataType, RegularizerType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
template<typename MatType, typename RegularizerType>
void Linear3DType<MatType, RegularizerType>::Forward(
const MatType& input, MatType& output)
{
typedef typename arma::Mat<eT> MatType;
typedef typename arma::Cube<eT> CubeType;
typedef typename arma::Cube<typename MatType::elem_type> CubeType;
if (input.n_rows % inSize != 0)
{
Log::Fatal << "Number of features in the input must be divisible by inSize."
<< std::endl;
}
const size_t nPoints = input.n_rows / inSize;
const size_t nPoints = input.n_rows / this->inputDimensions[0];
const size_t batchSize = input.n_cols;
output.set_size(outSize * nPoints, batchSize);
const CubeType inputTemp(const_cast<MatType&>(input).memptr(), inSize,
nPoints, batchSize, false, false);
const CubeType inputTemp(const_cast<MatType&>(input).memptr(),
this->inputDimensions[0], nPoints, batchSize, false, false);
for (size_t i = 0; i < batchSize; ++i)
{
@@ -139,16 +117,13 @@ void Linear3D<InputDataType, OutputDataType, RegularizerType>::Forward(
}
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
template<typename eT>
void Linear3D<InputDataType, OutputDataType, RegularizerType>::Backward(
const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g)
template<typename MatType, typename RegularizerType>
void Linear3DType<MatType, RegularizerType>::Backward(
const MatType& /* input */,
const MatType& gy,
MatType& g)
{
typedef typename arma::Mat<eT> MatType;
typedef typename arma::Cube<eT> CubeType;
typedef typename arma::Cube<typename MatType::elem_type> CubeType;
if (gy.n_rows % outSize != 0)
{
@@ -162,8 +137,6 @@ void Linear3D<InputDataType, OutputDataType, RegularizerType>::Backward(
const CubeType gyTemp(const_cast<MatType&>(gy).memptr(), outSize,
nPoints, batchSize, false, false);
g.set_size(inSize * nPoints, batchSize);
for (size_t i = 0; i < gyTemp.n_slices; ++i)
{
// Shape of weight : (outSize, inSize).
@@ -172,29 +145,26 @@ void Linear3D<InputDataType, OutputDataType, RegularizerType>::Backward(
}
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
template<typename eT>
void Linear3D<InputDataType, OutputDataType, RegularizerType>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient)
template<typename MatType, typename RegularizerType>
void Linear3DType<MatType, RegularizerType>::Gradient(
const MatType& input,
const MatType& error,
MatType& gradient)
{
typedef typename arma::Mat<eT> MatType;
typedef typename arma::Cube<eT> CubeType;
typedef typename arma::Cube<typename MatType::elem_type> CubeType;
if (error.n_rows % outSize != 0)
Log::Fatal << "Propagated error matrix has invalid dimension!" << std::endl;
const size_t nPoints = input.n_rows / inSize;
const size_t nPoints = input.n_rows / this->inputDimensions[0];
const size_t batchSize = input.n_cols;
const CubeType inputTemp(const_cast<MatType&>(input).memptr(), inSize,
nPoints, batchSize, false, false);
const CubeType inputTemp(const_cast<MatType&>(input).memptr(),
this->inputDimensions[0], nPoints, batchSize, false, false);
const CubeType errorTemp(const_cast<MatType&>(error).memptr(), outSize,
nPoints, batchSize, false, false);
CubeType dW(outSize, inSize, batchSize);
CubeType dW(outSize, this->inputDimensions[0], batchSize);
for (size_t i = 0; i < batchSize; ++i)
{
// Shape of errorTemp : (outSize, nPoints, batchSize).
@@ -202,8 +172,6 @@ void Linear3D<InputDataType, OutputDataType, RegularizerType>::Gradient(
dW.slice(i) = errorTemp.slice(i) * inputTemp.slice(i).t();
}
gradient.set_size(arma::size(weights));
gradient.submat(0, 0, weight.n_elem - 1, 0)
= arma::vectorise(arma::sum(dW, 2));
@@ -213,19 +181,27 @@ void Linear3D<InputDataType, OutputDataType, RegularizerType>::Gradient(
regularizer.Evaluate(weights, gradient);
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
template<typename MatType, typename RegularizerType>
void Linear3DType<
MatType, RegularizerType
>::ComputeOutputDimensions()
{
// The Linear3D layer shares weights for each row of the input, and
// duplicates it across the columns. Thus, we only change the number of
// rows.
this->outputDimensions = this->inputDimensions;
this->outputDimensions[0] = outSize;
}
template<typename MatType, typename RegularizerType>
template<typename Archive>
void Linear3D<InputDataType, OutputDataType, RegularizerType>::serialize(
void Linear3DType<MatType, RegularizerType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(CEREAL_NVP(inSize));
ar(CEREAL_NVP(outSize));
ar(cereal::base_class<Layer<MatType>>(this));
// This is inefficient, but we have to allocate this memory so that
// WeightSetVisitor gets the right size.
if (cereal::is_loading<Archive>())
weights.set_size(outSize * inSize + outSize, 1);
ar(CEREAL_NVP(outSize));
ar(CEREAL_NVP(regularizer));
}
} // namespace ann
src/mlpack/methods/ann/layer/linear_impl.hpp
+70 -65
View File
@@ -19,136 +19,141 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
Linear<InputDataType, OutputDataType, RegularizerType>::Linear() :
template<typename MatType, typename RegularizerType>
LinearType<MatType, RegularizerType>::LinearType() :
Layer<MatType>(),
inSize(0),
outSize(0)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
Linear<InputDataType, OutputDataType, RegularizerType>::Linear(
const size_t inSize,
template<typename MatType, typename RegularizerType>
LinearType<MatType, RegularizerType>::LinearType(
const size_t outSize,
RegularizerType regularizer) :
inSize(inSize),
Layer<MatType>(),
inSize(0), // This will be computed in ComputeOutputDimensions().
outSize(outSize),
regularizer(regularizer)
{
weights.set_size(WeightSize(), 1);
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
Linear<InputDataType, OutputDataType, RegularizerType>::Linear(
const Linear& layer) :
// Copy constructor.
template<typename MatType, typename RegularizerType>
LinearType<MatType, RegularizerType>::LinearType(const LinearType& layer) :
Layer<MatType>(layer),
inSize(layer.inSize),
outSize(layer.outSize),
weights(layer.weights),
regularizer(layer.regularizer)
{
// Nothing to do here.
// Nothing else to do.
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
Linear<InputDataType, OutputDataType, RegularizerType>::Linear(
Linear&& layer) :
inSize(0),
outSize(0),
weights(std::move(layer.weights)),
// Move constructor.
template<typename MatType, typename RegularizerType>
LinearType<MatType, RegularizerType>::LinearType(LinearType&& layer) :
Layer<MatType>(std::move(layer)),
inSize(std::move(layer.inSize)),
outSize(std::move(layer.outSize)),
regularizer(std::move(layer.regularizer))
{
// Nothing to do here.
// Nothing else to do.
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
Linear<InputDataType, OutputDataType, RegularizerType>&
Linear<InputDataType, OutputDataType, RegularizerType>::
operator=(const Linear& layer)
template<typename MatType, typename RegularizerType>
LinearType<MatType, RegularizerType>&
LinearType<MatType, RegularizerType>::operator=(const LinearType& layer)
{
if (this != &layer)
if (&layer != this)
{
Layer<MatType>::operator=(layer);
inSize = layer.inSize;
outSize = layer.outSize;
weights = layer.weights;
regularizer = layer.regularizer;
}
return *this;
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
Linear<InputDataType, OutputDataType, RegularizerType>&
Linear<InputDataType, OutputDataType, RegularizerType>::
operator=(Linear&& layer)
template<typename MatType, typename RegularizerType>
LinearType<MatType, RegularizerType>&
LinearType<MatType, RegularizerType>::operator=(
LinearType&& layer)
{
if (this != &layer)
if (&layer != this)
{
inSize = layer.inSize;
outSize = layer.outSize;
weights = std::move(layer.weights);
Layer<MatType>::operator=(std::move(layer));
inSize = std::move(layer.inSize);
outSize = std::move(layer.outSize);
regularizer = std::move(layer.regularizer);
}
return *this;
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
void Linear<InputDataType, OutputDataType, RegularizerType>::Reset()
template<typename MatType, typename RegularizerType>
void LinearType<MatType, RegularizerType>::SetWeights(
typename MatType::elem_type* weightsPtr)
{
weight = arma::mat(weights.memptr(), outSize, inSize, false, false);
bias = arma::mat(weights.memptr() + weight.n_elem,
outSize, 1, false, false);
MakeAlias(weights, weightsPtr, outSize * inSize + outSize, 1);
MakeAlias(weight, weightsPtr, outSize, inSize);
MakeAlias(bias, weightsPtr + weight.n_elem, outSize, 1);
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
template<typename eT>
void Linear<InputDataType, OutputDataType, RegularizerType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
template<typename MatType, typename RegularizerType>
void LinearType<MatType, RegularizerType>::Forward(
const MatType& input, MatType& output)
{
output = weight * input;
output.each_col() += bias;
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
template<typename eT>
void Linear<InputDataType, OutputDataType, RegularizerType>::Backward(
const arma::Mat<eT>& /* input */, const arma::Mat<eT>& gy, arma::Mat<eT>& g)
template<typename MatType, typename RegularizerType>
void LinearType<MatType, RegularizerType>::Backward(
const MatType& /* input */, const MatType& gy, MatType& g)
{
g = weight.t() * gy;
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
template<typename eT>
void Linear<InputDataType, OutputDataType, RegularizerType>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient)
template<typename MatType, typename RegularizerType>
void LinearType<MatType, RegularizerType>::Gradient(
const MatType& input,
const MatType& error,
MatType& gradient)
{
gradient.submat(0, 0, weight.n_elem - 1, 0) = arma::vectorise(
error * input.t());
gradient.submat(weight.n_elem, 0, gradient.n_elem - 1, 0) =
arma::sum(error, 1);
regularizer.Evaluate(weights, gradient);
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
template<typename MatType, typename RegularizerType>
void LinearType<MatType, RegularizerType>::ComputeOutputDimensions()
{
inSize = this->inputDimensions[0];
for (size_t i = 1; i < this->inputDimensions.size(); ++i)
inSize *= this->inputDimensions[i];
this->outputDimensions = std::vector<size_t>(this->inputDimensions.size(),
1);
// The Linear layer flattens its input.
this->outputDimensions[0] = outSize;
}
template<typename MatType, typename RegularizerType>
template<typename Archive>
void Linear<InputDataType, OutputDataType, RegularizerType>::serialize(
void LinearType<MatType, RegularizerType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(cereal::base_class<Layer<MatType>>(this));
ar(CEREAL_NVP(inSize));
ar(CEREAL_NVP(outSize));
ar(CEREAL_NVP(weights));
ar(CEREAL_NVP(regularizer));
}
} // namespace ann
src/mlpack/methods/ann/layer/linear_no_bias.hpp
+46 -91
View File
@@ -16,7 +16,7 @@
#include <mlpack/prereqs.hpp>
#include <mlpack/methods/ann/regularizer/no_regularizer.hpp>
#include "layer_types.hpp"
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -25,48 +25,50 @@ namespace ann /** Artificial Neural Network. */ {
* Implementation of the LinearNoBias class. The LinearNoBias class represents a
* single layer of a neural network.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam MatType Matrix representation to accept as input and use for
* computation.
* @tparam RegularizerType Type of the regularizer to be used (Default no
* regularizer).
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat,
template<
typename MatType = arma::mat,
typename RegularizerType = NoRegularizer
>
class LinearNoBias
class LinearNoBiasType : public Layer<MatType>
{
public:
//! Create the LinearNoBias object.
LinearNoBias();
LinearNoBiasType();
/**
* Create the LinearNoBias object using the specified number of units.
*
* @param inSize The number of input units.
* @param outSize The number of output units.
* @param regularizer The regularizer to use, optional.
*/
LinearNoBias(const size_t inSize,
const size_t outSize,
RegularizerType regularizer = RegularizerType());
LinearNoBiasType(const size_t outSize,
RegularizerType regularizer = RegularizerType());
//! Clone the LinearNoBiasType object. This handles polymorphism correctly.
LinearNoBiasType* Clone() const { return new LinearNoBiasType(*this); }
//! Reset the layer parameter.
void SetWeights(typename MatType::elem_type* weightsPtr);
//! Copy constructor.
LinearNoBias(const LinearNoBias& layer);
LinearNoBiasType(const LinearNoBiasType& layer);
//! Move constructor.
LinearNoBias(LinearNoBias&&);
LinearNoBiasType(LinearNoBiasType&&);
//! Copy assignment operator.
LinearNoBias& operator=(const LinearNoBias& layer);
LinearNoBiasType& operator=(const LinearNoBiasType& layer);
//! Move assignment operator.
LinearNoBias& operator=(LinearNoBias&& layer);
LinearNoBiasType& operator=(LinearNoBiasType&& layer);
/*
* Reset the layer parameter.
*/
void Reset();
//! Virtual destructor.
virtual ~LinearNoBiasType() { }
/**
* Ordinary feed forward pass of a neural network, evaluating the function
@@ -75,8 +77,7 @@ class LinearNoBias
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of a neural network, calculating the function
@@ -87,69 +88,33 @@ class LinearNoBias
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
void Backward(const MatType& /* input */,
const MatType& gy,
MatType& g);
/*
/**
* Calculate the gradient using the output delta and the input activation.
*
* @param input The input parameter used for calculating the gradient.
* @param error The calculated error.
* @param gradient The calculated gradient.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient);
void Gradient(const MatType& input,
const MatType& error,
MatType& gradient);
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
const MatType& Parameters() const { return weight; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
MatType& Parameters() { return weight; }
//! Get the input parameter.
InputDataType const& InputParameter() const { return inputParameter; }
//! Modify the input parameter.
InputDataType& InputParameter() { return inputParameter; }
//! Get the number of weights in the layer.
size_t WeightSize() const { return inSize * outSize; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Compute the output dimensions of the layer using `InputDimensions()`.
void ComputeOutputDimensions();
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the input size.
size_t InputSize() const { return inSize; }
//! Get the output size.
size_t OutputSize() const { return outSize; }
//! Get the gradient.
OutputDataType const& Gradient() const { return gradient; }
//! Modify the gradient.
OutputDataType& Gradient() { return gradient; }
//! Get the size of the weights.
size_t WeightSize() const
{
return inSize * outSize;
}
//! Get the shape of the input.
size_t InputShape() const
{
return inSize;
}
/**
* Serialize the layer
*/
//! Serialize the layer.
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
@@ -160,27 +125,17 @@ class LinearNoBias
//! Locally-stored number of output units.
size_t outSize;
//! Locally-stored weight object.
OutputDataType weights;
//! Locally-stored weight parameter.
OutputDataType weight;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored gradient object.
OutputDataType gradient;
//! Locally-stored input parameter object.
InputDataType inputParameter;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
MatType weight;
//! Locally-stored regularizer object.
RegularizerType regularizer;
}; // class LinearNoBias
}; // class LinearNoBiasType
// Convenience typedefs.
// Standard Linear without bias layer using no regularization.
typedef LinearNoBiasType<arma::mat, NoRegularizer> LinearNoBias;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/linear_no_bias_impl.hpp
+64 -63
View File
@@ -19,135 +19,136 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
LinearNoBias<InputDataType, OutputDataType, RegularizerType>::LinearNoBias() :
template<typename MatType, typename RegularizerType>
LinearNoBiasType<MatType, RegularizerType>::LinearNoBiasType() :
Layer<MatType>(),
inSize(0),
outSize(0)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
LinearNoBias<InputDataType, OutputDataType, RegularizerType>::LinearNoBias(
const size_t inSize,
template<typename MatType, typename RegularizerType>
LinearNoBiasType<MatType, RegularizerType>::LinearNoBiasType(
const size_t outSize,
RegularizerType regularizer) :
inSize(inSize),
Layer<MatType>(),
inSize(0), // This will be set by ComputeOutputDimensions().
outSize(outSize),
regularizer(regularizer)
{
weights.set_size(WeightSize(), 1);
// Nothing to do.
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
LinearNoBias<InputDataType, OutputDataType, RegularizerType>::LinearNoBias(
const LinearNoBias& layer) :
template<typename MatType, typename RegularizerType>
LinearNoBiasType<MatType, RegularizerType>::LinearNoBiasType(
const LinearNoBiasType& layer) :
Layer<MatType>(layer),
inSize(layer.inSize),
outSize(layer.outSize),
weights(layer.weights),
regularizer(layer.regularizer)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
LinearNoBias<InputDataType, OutputDataType, RegularizerType>::LinearNoBias(
LinearNoBias&& layer) :
template<typename MatType, typename RegularizerType>
LinearNoBiasType<MatType, RegularizerType>::LinearNoBiasType(
LinearNoBiasType&& layer) :
Layer<MatType>(std::move(layer)),
inSize(0),
outSize(0),
weights(std::move(layer.weights)),
regularizer(std::move(layer.regularizer))
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
LinearNoBias<InputDataType, OutputDataType, RegularizerType>&
LinearNoBias<InputDataType, OutputDataType, RegularizerType>::
operator=(const LinearNoBias& layer)
template<typename MatType, typename RegularizerType>
LinearNoBiasType<MatType, RegularizerType>&
LinearNoBiasType<MatType, RegularizerType>::operator=(
const LinearNoBiasType& layer)
{
if (this != &layer)
{
Layer<MatType>::operator=(layer);
inSize = layer.inSize;
outSize = layer.outSize;
weights = layer.weights;
regularizer = layer.regularizer;
}
return *this;
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
LinearNoBias<InputDataType, OutputDataType, RegularizerType>&
LinearNoBias<InputDataType, OutputDataType, RegularizerType>::
operator=(LinearNoBias&& layer)
template<typename MatType, typename RegularizerType>
LinearNoBiasType<MatType, RegularizerType>&
LinearNoBiasType<MatType, RegularizerType>::operator=(
LinearNoBiasType&& layer)
{
if (this != &layer)
{
inSize = layer.inSize;
outSize = layer.outSize;
weights = std::move(layer.weights);
Layer<MatType>::operator=(std::move(layer));
inSize = std::move(layer.inSize);
outSize = std::move(layer.outSize);
regularizer = std::move(layer.regularizer);
}
return *this;
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
void LinearNoBias<InputDataType, OutputDataType, RegularizerType>::Reset()
template<typename MatType, typename RegularizerType>
void LinearNoBiasType<MatType, RegularizerType>::SetWeights(
typename MatType::elem_type* weightsPtr)
{
weight = arma::mat(weights.memptr(), outSize, inSize, false, false);
MakeAlias(weight, weightsPtr, outSize, inSize);
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
template<typename eT>
void LinearNoBias<InputDataType, OutputDataType, RegularizerType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
template<typename MatType, typename RegularizerType>
void LinearNoBiasType<MatType, RegularizerType>::Forward(
const MatType& input, MatType& output)
{
output = weight * input;
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
template<typename eT>
void LinearNoBias<InputDataType, OutputDataType, RegularizerType>::Backward(
const arma::Mat<eT>& /* input */, const arma::Mat<eT>& gy, arma::Mat<eT>& g)
template<typename MatType, typename RegularizerType>
void LinearNoBiasType<MatType, RegularizerType>::Backward(
const MatType& /* input */, const MatType& gy, MatType& g)
{
g = weight.t() * gy;
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
template<typename eT>
void LinearNoBias<InputDataType, OutputDataType, RegularizerType>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient)
template<typename MatType, typename RegularizerType>
void LinearNoBiasType<MatType, RegularizerType>::Gradient(
const MatType& input,
const MatType& error,
MatType& gradient)
{
gradient.submat(0, 0, weight.n_elem - 1, 0) = arma::vectorise(
error * input.t());
regularizer.Evaluate(weights, gradient);
regularizer.Evaluate(weight, gradient);
}
template<typename InputDataType, typename OutputDataType,
typename RegularizerType>
template<typename MatType, typename RegularizerType>
void LinearNoBiasType<MatType, RegularizerType>::ComputeOutputDimensions()
{
inSize = this->inputDimensions[0];
for (size_t i = 1; i < this->inputDimensions.size(); ++i)
inSize *= this->inputDimensions[i];
this->outputDimensions = std::vector<size_t>(this->inputDimensions.size(),
1);
this->outputDimensions[0] = outSize;
}
template<typename MatType, typename RegularizerType>
template<typename Archive>
void LinearNoBias<InputDataType, OutputDataType, RegularizerType>::serialize(
void LinearNoBiasType<MatType, RegularizerType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(cereal::base_class<Layer<MatType>>(this));
ar(CEREAL_NVP(inSize));
ar(CEREAL_NVP(outSize));
// This is inefficient, but necessary so that WeightSetVisitor sets the right
// size.
if (cereal::is_loading<Archive>())
weights.set_size(outSize * inSize, 1);
ar(CEREAL_NVP(regularizer));
}
} // namespace ann
src/mlpack/methods/ann/layer/log_softmax.hpp
+35 -36
View File
@@ -14,6 +14,8 @@
#include <mlpack/prereqs.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -24,22 +26,32 @@ namespace ann /** Artificial Neural Network. */ {
* (NegativeLogLikelihoodLayer), which expects that the input contains
* log-probabilities for each class.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class LogSoftMax
template <typename MatType = arma::mat>
class LogSoftMaxType : public Layer<MatType>
{
public:
/**
* Create the LogSoftmax object.
* Create the LogSoftmax layer.
*/
LogSoftMax();
LogSoftMaxType();
//! Clone the LogSoftMaxType object. This handles polymorphism correctly.
LogSoftMaxType* Clone() const { return new LogSoftMaxType(*this); }
// Virtual destructor.
virtual ~LogSoftMaxType() { }
//! Copy the given LogSoftMaxType.
LogSoftMaxType(const LogSoftMaxType& other);
//! Take ownership of the given LogSoftMaxType.
LogSoftMaxType(LogSoftMaxType&& other);
//! Copy the given LogSoftMaxType.
LogSoftMaxType& operator=(const LogSoftMaxType& other);
//! Take ownership of the given LogSoftMaxType.
LogSoftMaxType& operator=(LogSoftMaxType&& other);
/**
* Ordinary feed forward pass of a neural network, evaluating the function
@@ -48,8 +60,7 @@ class LogSoftMax
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename InputType, typename OutputType>
void Forward(const InputType& input, OutputType& output);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of a neural network, calculating the function
@@ -60,34 +71,22 @@ class LogSoftMax
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& input,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
void Backward(const MatType& input, const MatType& gy, MatType& g);
//! Get the output parameter.
OutputDataType& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
InputDataType& Delta() const { return delta; }
//! Modify the delta.
InputDataType& Delta() { return delta; }
/**
* Serialize the layer.
*/
template<typename Archive>
void serialize(Archive& /* ar */, const uint32_t /* version */);
void serialize(Archive& ar, const uint32_t /* version */)
{
ar(cereal::base_class<Layer<MatType>>(this));
// Nothing to do.
}
private:
//! Locally-stored delta object.
OutputDataType delta;
}; // class LogSoftmaxType
//! Locally-stored output parameter object.
OutputDataType outputParameter;
}; // class LogSoftmax
// Convenience typedefs.
// Standard Linear layer using no regularization.
typedef LogSoftMaxType<arma::mat> LogSoftMax;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/log_softmax_impl.hpp
+48 -22
View File
@@ -18,18 +18,54 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
LogSoftMax<InputDataType, OutputDataType>::LogSoftMax()
template<typename MatType>
LogSoftMaxType<MatType>::LogSoftMaxType()
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
void LogSoftMax<InputDataType, OutputDataType>::Forward(
const InputType& input, OutputType& output)
template<typename MatType>
LogSoftMaxType<MatType>::LogSoftMaxType(const LogSoftMaxType& other) :
Layer<MatType>(other)
{
arma::mat maxInput = arma::repmat(arma::max(input), input.n_rows, 1);
// Nothing to do here.
}
template<typename MatType>
LogSoftMaxType<MatType>::LogSoftMaxType(LogSoftMaxType&& other) :
Layer<MatType>(std::move(other))
{
// Nothing to do here.
}
template<typename MatType>
LogSoftMaxType<MatType>&
LogSoftMaxType<MatType>::operator=(const LogSoftMaxType& other)
{
if (&other != this)
{
Layer<MatType>::operator=(other);
}
return *this;
}
template<typename MatType>
LogSoftMaxType<MatType>&
LogSoftMaxType<MatType>::operator=(LogSoftMaxType&& other)
{
if (&other != this)
{
Layer<MatType>::operator=(std::move(other));
}
return *this;
}
template<typename MatType>
void LogSoftMaxType<MatType>::Forward(const MatType& input, MatType& output)
{
MatType maxInput = arma::repmat(arma::max(input), input.n_rows, 1);
output = (maxInput - input);
// Approximation of the base-e exponential function. The acuracy however is
@@ -61,25 +97,15 @@ void LogSoftMax<InputDataType, OutputDataType>::Forward(
output = input - maxInput;
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void LogSoftMax<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& input,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g)
template<typename MatType>
void LogSoftMaxType<MatType>::Backward(
const MatType& input,
const MatType& gy,
MatType& g)
{
g = arma::exp(input) + gy;
}
template<typename InputDataType, typename OutputDataType>
template<typename Archive>
void LogSoftMax<InputDataType, OutputDataType>::serialize(
Archive& /* ar */,
const uint32_t /* version */)
{
// Nothing to do here.
}
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/lstm.hpp
+98 -177
View File
@@ -15,6 +15,8 @@
#include <mlpack/prereqs.hpp>
#include <limits>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -50,43 +52,43 @@ namespace ann /** Artificial Neural Network. */ {
* \see FastLSTM for a faster LSTM version which combines the calculation of the
* input, forget, output gates and hidden state in a single step.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class LSTM
template<typename MatType = arma::mat>
class LSTMType : public RecurrentLayer<MatType>
{
public:
//! Create the LSTM object.
LSTM();
LSTMType();
/**
* Create the LSTM layer object using the specified parameters.
*
* @param inSize The number of input units.
* @param outSize The number of output units.
* @param rho Maximum number of steps to backpropagate through time (BPTT).
*/
LSTM(const size_t inSize,
const size_t outSize,
const size_t rho = std::numeric_limits<size_t>::max());
LSTMType(const size_t outSize);
//! Copy constructor.
LSTM(const LSTM& layer);
//! Clone the LSTMType object. This handles polymorphism correctly.
LSTMType* Clone() const { return new LSTMType(*this); }
//! Move constructor.
LSTM(LSTM&&);
//! Copy the given LSTMType object.
LSTMType(const LSTMType& other);
//! Take ownership of the given LSTMType object's data.
LSTMType(LSTMType&& other);
//! Copy the given LSTMType object.
LSTMType& operator=(const LSTMType& other);
//! Take ownership of the given LSTMType object's data.
LSTMType& operator=(LSTMType&& other);
//! Copy assignment operator.
LSTM& operator=(const LSTM& layer);
virtual ~LSTMType() { }
//! Move assignment operator.
LSTM& operator=(LSTM&& layer);
/**
* Reset the layer parameter. The method is called to
* assign the allocated memory to the internal learnable parameters.
*/
void SetWeights(typename MatType::elem_type* weightsPtr);
/**
* Ordinary feed-forward pass of a neural network, evaluating the function
@@ -95,23 +97,7 @@ class LSTM
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename InputType, typename OutputType>
void Forward(const InputType& input, OutputType& output);
/**
* Ordinary feed-forward pass of a neural network, evaluating the function
* f(x) by propagating the activity forward through f.
*
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
* @param cellState Cell state of the LSTM.
* @param useCellState Use the cellState passed in the LSTM cell.
*/
template<typename InputType, typename OutputType>
void Forward(const InputType& input,
OutputType& output,
OutputType& cellState,
bool useCellState = false);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of a neural network, calculating the function
@@ -122,23 +108,7 @@ class LSTM
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename InputType, typename ErrorType, typename GradientType>
void Backward(const InputType& input,
const ErrorType& gy,
GradientType& g);
/*
* Reset the layer parameter.
*/
void Reset();
/*
* Resets the cell to accept a new input. This breaks the BPTT chain starts a
* new one.
*
* @param size The current maximum number of steps through time.
*/
void ResetCell(const size_t size);
void Backward(const MatType& input, const MatType& gy, MatType& g);
/*
* Calculate the gradient using the output delta and the input activation.
@@ -147,56 +117,44 @@ class LSTM
* @param error The calculated error.
* @param gradient The calculated gradient.
*/
template<typename InputType, typename ErrorType, typename GradientType>
void Gradient(const InputType& input,
const ErrorType& error,
GradientType& gradient);
void Gradient(const MatType& input,
const MatType& error,
MatType& gradient);
//! Get the maximum number of steps to backpropagate through time (BPTT).
size_t Rho() const { return rho; }
//! Modify the maximum number of steps to backpropagate through time (BPTT).
size_t& Rho() { return rho; }
/**
* Reset the recurrent state of the LSTM layer, and allocate enough space to
* hold `bpttSteps` of previous passes with a batch size of `batchSize`.
*
* @param bpttSteps Number of steps of history to allocate space for.
* @param batchSize Batch size to prepare for.
*/
void ClearRecurrentState(const size_t bpttSteps, const size_t batchSize);
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
const MatType& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
MatType& Parameters() { return weights; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the gradient.
OutputDataType const& Gradient() const { return grad; }
//! Modify the gradient.
OutputDataType& Gradient() { return grad; }
//! Get the number of input units.
size_t InSize() const { return inSize; }
//! Get the number of output units.
size_t OutSize() const { return outSize; }
//! Get the size of the weights.
//! Get the total number of trainable parameters.
size_t WeightSize() const
{
return (4 * outSize * inSize + 7 * outSize + 4 * outSize * outSize);
}
//! Get the shape of the input.
size_t InputShape() const
//! Given a properly set InputDimensions(), compute the output dimensions.
void ComputeOutputDimensions()
{
return inSize;
inSize = std::accumulate(this->inputDimensions.begin(),
this->inputDimensions.end(), 0);
this->outputDimensions = std::vector<size_t>(this->inputDimensions.size(),
1);
// The LSTM layer flattens its input.
this->outputDimensions[0] = outSize;
}
/**
* Serialize the layer
* Serialize the layer.
*/
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
@@ -208,148 +166,111 @@ class LSTM
//! Locally-stored number of output units.
size_t outSize;
//! Number of steps to backpropagate through time (BPTT).
size_t rho;
//! Locally-stored number of forward steps.
size_t forwardStep;
//! Locally-stored number of backward steps.
size_t backwardStep;
//! Locally-stored number of gradient steps.
size_t gradientStep;
//! Locally-stored weight object.
OutputDataType weights;
//! Locally-stored previous output.
OutputDataType prevOutput;
//! Locally-stored batch size.
size_t batchSize;
//! Current batch step, alias for batchSize - 1.
size_t batchStep;
//! Current gradient step to keep track of the backpropagate through time
//! step.
size_t gradientStepIdx;
//! Locally-stored cell activation error.
OutputDataType cellActivationError;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored gradient object.
OutputDataType grad;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
MatType weights;
//! Weights between the output and input gate.
OutputDataType output2GateInputWeight;
MatType output2GateInputWeight;
//! Weights between the input and gate.
OutputDataType input2GateInputWeight;
MatType input2GateInputWeight;
//! Bias between the input and input gate.
OutputDataType input2GateInputBias;
MatType input2GateInputBias;
//! Weights between the cell and input gate.
OutputDataType cell2GateInputWeight;
MatType cell2GateInputWeight;
//! Weights between the output and forget gate.
OutputDataType output2GateForgetWeight;
MatType output2GateForgetWeight;
//! Weights between the input and gate.
OutputDataType input2GateForgetWeight;
MatType input2GateForgetWeight;
//! Bias between the input and gate.
OutputDataType input2GateForgetBias;
MatType input2GateForgetBias;
//! Bias between the input and gate.
OutputDataType cell2GateForgetWeight;
MatType cell2GateForgetWeight;
//! Weights between the output and gate.
OutputDataType output2GateOutputWeight;
MatType output2GateOutputWeight;
//! Weights between the input and gate.
OutputDataType input2GateOutputWeight;
MatType input2GateOutputWeight;
//! Bias between the input and gate.
OutputDataType input2GateOutputBias;
MatType input2GateOutputBias;
//! Weights between cell and output gate.
OutputDataType cell2GateOutputWeight;
MatType cell2GateOutputWeight;
// Below here are recurrent state matrices.
//! Locally-stored input gate parameter.
OutputDataType inputGate;
MatType inputGate;
//! Locally-stored forget gate parameter.
OutputDataType forgetGate;
MatType forgetGate;
//! Locally-stored hidden layer parameter.
OutputDataType hiddenLayer;
MatType hiddenLayer;
//! Locally-stored output gate parameter.
OutputDataType outputGate;
//! Locally-stored input gate activation.
OutputDataType inputGateActivation;
//! Locally-stored forget gate activation.
OutputDataType forgetGateActivation;
//! Locally-stored output gate activation.
OutputDataType outputGateActivation;
//! Locally-stored hidden layer activation.
OutputDataType hiddenLayerActivation;
MatType outputGate;
//! Locally-stored input to hidden weight.
OutputDataType input2HiddenWeight;
MatType input2HiddenWeight;
//! Locally-stored input to hidden bias.
OutputDataType input2HiddenBias;
MatType input2HiddenBias;
//! Locally-stored output to hidden weight.
OutputDataType output2HiddenWeight;
MatType output2HiddenWeight;
//! Locally-stored cell parameter.
OutputDataType cell;
arma::Cube<typename MatType::elem_type> cell;
// These members store recurrent state.
//! Locally-stored input gate activation.
arma::Cube<typename MatType::elem_type> inputGateActivation;
//! Locally-stored forget gate activation.
arma::Cube<typename MatType::elem_type> forgetGateActivation;
//! Locally-stored output gate activation.
arma::Cube<typename MatType::elem_type> outputGateActivation;
//! Locally-stored hidden layer activation.
arma::Cube<typename MatType::elem_type> hiddenLayerActivation;
//! Locally-stored cell activation error.
OutputDataType cellActivation;
arma::Cube<typename MatType::elem_type> cellActivation;
//! Locally-stored forget gate error.
OutputDataType forgetGateError;
MatType forgetGateError;
//! Locally-stored output gate error.
OutputDataType outputGateError;
//! Locally-stored previous error.
OutputDataType prevError;
MatType outputGateError;
//! Locally-stored output parameters.
OutputDataType outParameter;
arma::Cube<typename MatType::elem_type> outParameter;
//! Locally-stored input cell error parameter.
OutputDataType inputCellError;
MatType inputCellError;
//! Locally-stored input gate error.
OutputDataType inputGateError;
MatType inputGateError;
//! Locally-stored hidden layer error.
OutputDataType hiddenError;
MatType hiddenError;
}; // class LSTMType
//! Locally-stored current rho size.
size_t rhoSize;
// Convenience typedefs.
//! Current backpropagate through time steps.
size_t bpttSteps;
}; // class LSTM
// Standard LSTM layer.
typedef LSTMType<arma::mat> LSTM;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/lstm_impl.hpp
+211 -359
View File
@@ -18,430 +18,296 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
LSTM<InputDataType, OutputDataType>::LSTM()
template<typename MatType>
LSTMType<MatType>::LSTMType() :
RecurrentLayer<MatType>(),
outSize(0)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
LSTM<InputDataType, OutputDataType>::LSTM(
const LSTM& layer) :
inSize(layer.inSize),
outSize(layer.outSize),
rho(layer.rho),
forwardStep(layer.forwardStep),
backwardStep(layer.backwardStep),
gradientStep(layer.gradientStep),
weights(layer.weights),
batchSize(layer.batchSize),
batchStep(layer.batchStep),
gradientStepIdx(layer.gradientStepIdx),
rhoSize(layer.rho),
bpttSteps(layer.bpttSteps)
template<typename MatType>
LSTMType<MatType>::LSTMType(const size_t outSize) :
RecurrentLayer<MatType>(),
outSize(outSize)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
LSTM<InputDataType, OutputDataType>::LSTM(
LSTM&& layer) :
inSize(std::move(layer.inSize)),
outSize(std::move(layer.outSize)),
rho(std::move(layer.rho)),
forwardStep(std::move(layer.forwardStep)),
backwardStep(std::move(layer.backwardStep)),
gradientStep(std::move(layer.gradientStep)),
weights(std::move(layer.weights)),
batchSize(std::move(layer.batchSize)),
batchStep(std::move(layer.batchStep)),
gradientStepIdx(std::move(layer.gradientStepIdx)),
rhoSize(std::move(layer.rho)),
bpttSteps(std::move(layer.bpttSteps))
template<typename MatType>
LSTMType<MatType>::LSTMType(const LSTMType& layer) :
RecurrentLayer<MatType>(layer)
{
// Nothing to do here.
}
template <typename InputDataType, typename OutputDataType>
LSTM<InputDataType, OutputDataType>&
LSTM<InputDataType, OutputDataType> :: operator=(const LSTM& layer)
template<typename MatType>
LSTMType<MatType>::LSTMType(LSTMType&& layer) :
RecurrentLayer<MatType>(std::move(layer))
{
// Nothing to do here.
}
template<typename MatType>
LSTMType<MatType>& LSTMType<MatType>::operator=(const LSTMType& layer)
{
if (this != &layer)
{
inSize = layer.inSize;
outSize = layer.outSize;
rho = layer.rho;
forwardStep = layer.forwardStep;
backwardStep = layer.backwardStep;
gradientStep = layer.gradientStep;
weights = layer.weights;
batchSize = layer.batchSize;
batchStep = layer.batchStep;
gradientStepIdx = layer.gradientStepIdx;
grad = layer.grad;
rhoSize = layer.rho;
bpttSteps = layer.bpttSteps;
RecurrentLayer<MatType>::operator=(layer);
}
return *this;
}
template <typename InputDataType, typename OutputDataType>
LSTM<InputDataType, OutputDataType>&
LSTM<InputDataType, OutputDataType> :: operator=(LSTM&& layer)
template<typename MatType>
LSTMType<MatType>& LSTMType<MatType>::operator=(LSTMType&& layer)
{
if (this != &layer)
{
inSize = std::move(layer.inSize);
outSize = std::move(layer.outSize);
rho = std::move(layer.rho);
forwardStep = std::move(layer.forwardStep);
backwardStep = std::move(layer.backwardStep);
gradientStep = std::move(layer.gradientStep);
weights = std::move(layer.weights);
batchSize = std::move(layer.batchSize);
batchStep = std::move(layer.batchStep);
gradientStepIdx = std::move(layer.gradientStepIdx);
grad = std::move(layer.grad);
rhoSize = std::move(layer.rho);
bpttSteps = std::move(layer.bpttSteps);
RecurrentLayer<MatType>::operator=(std::move(layer));
}
return *this;
}
template <typename InputDataType, typename OutputDataType>
LSTM<InputDataType, OutputDataType>::LSTM(
const size_t inSize, const size_t outSize, const size_t rho) :
inSize(inSize),
outSize(outSize),
rho(rho),
forwardStep(0),
backwardStep(0),
gradientStep(0),
batchSize(0),
batchStep(0),
gradientStepIdx(0),
rhoSize(rho),
bpttSteps(0)
template<typename MatType>
void LSTMType<MatType>::ClearRecurrentState(
const size_t bpttSteps, const size_t batchSize)
{
weights.set_size(WeightSize(), 1);
}
template<typename InputDataType, typename OutputDataType>
void LSTM<InputDataType, OutputDataType>::ResetCell(const size_t size)
{
if (size == std::numeric_limits<size_t>::max())
return;
rhoSize = size;
if (batchSize == 0)
return;
bpttSteps = std::min(rho, rhoSize);
forwardStep = 0;
gradientStepIdx = 0;
backwardStep = batchSize * size - 1;
gradientStep = batchSize * size - 1;
const size_t rhoBatchSize = size * batchSize;
// Make sure all of the different matrices we will use to hold parameters are
// at least as large as we need.
inputGate.set_size(outSize, rhoBatchSize);
forgetGate.set_size(outSize, rhoBatchSize);
hiddenLayer.set_size(outSize, rhoBatchSize);
outputGate.set_size(outSize, rhoBatchSize);
inputGate.set_size(outSize, batchSize);
forgetGate.set_size(outSize, batchSize);
hiddenLayer.set_size(outSize, batchSize);
outputGate.set_size(outSize, batchSize);
inputGateActivation.set_size(outSize, rhoBatchSize);
forgetGateActivation.set_size(outSize, rhoBatchSize);
outputGateActivation.set_size(outSize, rhoBatchSize);
hiddenLayerActivation.set_size(outSize, rhoBatchSize);
inputGateActivation.set_size(outSize, batchSize, bpttSteps);
forgetGateActivation.set_size(outSize, batchSize, bpttSteps);
outputGateActivation.set_size(outSize, batchSize, bpttSteps);
hiddenLayerActivation.set_size(outSize, batchSize, bpttSteps);
cellActivation.set_size(outSize, rhoBatchSize);
prevError.set_size(4 * outSize, batchSize);
cellActivation.set_size(outSize, batchSize, bpttSteps);
outParameter.set_size(outSize, batchSize, bpttSteps);
// Now reset recurrent values to 0.
cell.zeros(outSize, size * batchSize);
outParameter.zeros(outSize, (size + 1) * batchSize);
cell.zeros(outSize, batchSize, bpttSteps);
}
template<typename InputDataType, typename OutputDataType>
void LSTM<InputDataType, OutputDataType>::Reset()
template<typename MatType>
void LSTMType<MatType>::SetWeights(
typename MatType::elem_type* weightsPtr)
{
// Set the weight parameter for the output gate.
input2GateOutputWeight = OutputDataType(weights.memptr(), outSize, inSize,
false, false);
input2GateOutputBias = OutputDataType(weights.memptr() +
input2GateOutputWeight.n_elem, outSize, 1, false, false);
size_t offset = input2GateOutputWeight.n_elem + input2GateOutputBias.n_elem;
MakeAlias(input2GateOutputWeight, weightsPtr, outSize, inSize);
size_t offset = input2GateOutputWeight.n_elem;
MakeAlias(input2GateOutputBias, weightsPtr + offset, outSize, 1);
offset += input2GateOutputBias.n_elem;
// Set the weight parameter for the forget gate.
input2GateForgetWeight = OutputDataType(weights.memptr() + offset,
outSize, inSize, false, false);
input2GateForgetBias = OutputDataType(weights.memptr() +
offset + input2GateForgetWeight.n_elem, outSize, 1, false, false);
offset += input2GateForgetWeight.n_elem + input2GateForgetBias.n_elem;
MakeAlias(input2GateForgetWeight, weightsPtr + offset, outSize, inSize);
offset += input2GateForgetWeight.n_elem;
MakeAlias(input2GateForgetBias, weightsPtr + offset, outSize, 1);
offset += input2GateForgetBias.n_elem;
// Set the weight parameter for the input gate.
input2GateInputWeight = OutputDataType(weights.memptr() +
offset, outSize, inSize, false, false);
input2GateInputBias = OutputDataType(weights.memptr() +
offset + input2GateInputWeight.n_elem, outSize, 1, false, false);
offset += input2GateInputWeight.n_elem + input2GateInputBias.n_elem;
MakeAlias(input2GateInputWeight, weightsPtr + offset, outSize, inSize);
offset += input2GateInputWeight.n_elem;
MakeAlias(input2GateInputBias, weightsPtr + offset, outSize, 1);
offset += input2GateInputBias.n_elem;
// Set the weight parameter for the hidden gate.
input2HiddenWeight = OutputDataType(weights.memptr() +
offset, outSize, inSize, false, false);
input2HiddenBias = OutputDataType(weights.memptr() +
offset + input2HiddenWeight.n_elem, outSize, 1, false, false);
offset += input2HiddenWeight.n_elem + input2HiddenBias.n_elem;
MakeAlias(input2HiddenWeight, weightsPtr + offset, outSize, inSize);
offset += input2HiddenWeight.n_elem;
MakeAlias(input2HiddenBias, weightsPtr + offset, outSize, 1);
offset += input2HiddenBias.n_elem;
// Set the weight parameter for the output multiplication.
output2GateOutputWeight = OutputDataType(weights.memptr() +
offset, outSize, outSize, false, false);
MakeAlias(output2GateOutputWeight, weightsPtr + offset, outSize, outSize);
offset += output2GateOutputWeight.n_elem;
// Set the weight parameter for the output multiplication.
output2GateForgetWeight = OutputDataType(weights.memptr() +
offset, outSize, outSize, false, false);
MakeAlias(output2GateForgetWeight, weightsPtr + offset, outSize, outSize);
offset += output2GateForgetWeight.n_elem;
// Set the weight parameter for the input multiplication.
output2GateInputWeight = OutputDataType(weights.memptr() +
offset, outSize, outSize, false, false);
MakeAlias(output2GateInputWeight, weightsPtr + offset, outSize, outSize);
offset += output2GateInputWeight.n_elem;
// Set the weight parameter for the hidden multiplication.
output2HiddenWeight = OutputDataType(weights.memptr() +
offset, outSize, outSize, false, false);
MakeAlias(output2HiddenWeight, weightsPtr + offset, outSize, outSize);
offset += output2HiddenWeight.n_elem;
// Set the weight parameter for the cell multiplication.
cell2GateOutputWeight = OutputDataType(weights.memptr() +
offset, outSize, 1, false, false);
MakeAlias(cell2GateOutputWeight, weightsPtr + offset, outSize, 1);
offset += cell2GateOutputWeight.n_elem;
// Set the weight parameter for the cell - forget gate multiplication.
cell2GateForgetWeight = OutputDataType(weights.memptr() +
offset, outSize, 1, false, false);
MakeAlias(cell2GateForgetWeight, weightsPtr + offset, outSize, 1);
offset += cell2GateOutputWeight.n_elem;
// Set the weight parameter for the cell - input gate multiplication.
cell2GateInputWeight = OutputDataType(weights.memptr() +
offset, outSize, 1, false, false);
MakeAlias(cell2GateInputWeight, weightsPtr + offset, outSize, 1);
}
// Forward when cellState is not needed.
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
void LSTM<InputDataType, OutputDataType>::Forward(
const InputType& input, OutputType& output)
template<typename MatType>
void LSTMType<MatType>::Forward(const MatType& input, MatType& output)
{
//! Locally-stored cellState.
OutputType cellState;
Forward(input, output, cellState, false);
}
// Convenience alias.
const size_t batchSize = input.n_cols;
// Forward when cellState is needed overloaded LSTM::Forward().
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
void LSTM<InputDataType, OutputDataType>::Forward(const InputType& input,
OutputType& output,
OutputType& cellState,
bool useCellState)
{
// Check if the batch size changed, the number of cols is defines the input
// batch size.
if (input.n_cols != batchSize)
inputGate = input2GateInputWeight * input;
if (this->HasPreviousStep())
{
batchSize = input.n_cols;
batchStep = batchSize - 1;
ResetCell(rhoSize);
inputGate +=
output2GateInputWeight * outParameter.slice(this->PreviousStep());
}
inputGate.each_col() += input2GateInputBias;
forgetGate = input2GateForgetWeight * input;
if (this->HasPreviousStep())
{
forgetGate += output2GateForgetWeight * outParameter.slice(
this->PreviousStep());
}
forgetGate.each_col() += input2GateForgetBias;
if (this->HasPreviousStep())
{
inputGate += arma::repmat(cell2GateInputWeight, 1, batchSize) %
cell.slice(this->PreviousStep());
forgetGate += arma::repmat(cell2GateForgetWeight, 1, batchSize) %
cell.slice(this->PreviousStep());
}
inputGate.cols(forwardStep, forwardStep + batchStep) = input2GateInputWeight *
input + output2GateInputWeight * outParameter.cols(forwardStep,
forwardStep + batchStep);
inputGate.cols(forwardStep, forwardStep + batchStep).each_col() +=
input2GateInputBias;
inputGateActivation.slice(this->CurrentStep()) =
1.0 / (1.0 + arma::exp(-inputGate));
forgetGateActivation.slice(this->CurrentStep()) =
1.0 / (1.0 + arma::exp(-forgetGate));
forgetGate.cols(forwardStep, forwardStep + batchStep) = input2GateForgetWeight
* input + output2GateForgetWeight * outParameter.cols(
forwardStep, forwardStep + batchStep);
forgetGate.cols(forwardStep, forwardStep + batchStep).each_col() +=
input2GateForgetBias;
if (forwardStep > 0)
hiddenLayer = input2HiddenWeight * input;
if (this->HasPreviousStep())
{
if (useCellState)
{
if (!cellState.is_empty())
{
cell.cols(forwardStep - batchSize,
forwardStep - batchSize + batchStep) = cellState;
}
else
{
throw std::runtime_error("Cell parameter is empty.");
}
}
inputGate.cols(forwardStep, forwardStep + batchStep) +=
arma::repmat(cell2GateInputWeight, 1, batchSize) %
cell.cols(forwardStep - batchSize, forwardStep - batchSize + batchStep);
forgetGate.cols(forwardStep, forwardStep + batchStep) +=
arma::repmat(cell2GateForgetWeight, 1, batchSize) %
cell.cols(forwardStep - batchSize, forwardStep - batchSize + batchStep);
hiddenLayer += output2HiddenWeight *
outParameter.slice(this->PreviousStep());
}
hiddenLayer.each_col() += input2HiddenBias;
inputGateActivation.cols(forwardStep, forwardStep + batchStep) = 1.0 /
(1 + arma::exp(-inputGate.cols(forwardStep, forwardStep + batchStep)));
hiddenLayerActivation.slice(this->CurrentStep()) = arma::tanh(hiddenLayer);
forgetGateActivation.cols(forwardStep, forwardStep + batchStep) = 1.0 /
(1 + arma::exp(-forgetGate.cols(forwardStep, forwardStep + batchStep)));
hiddenLayer.cols(forwardStep, forwardStep + batchStep) = input2HiddenWeight *
input + output2HiddenWeight * outParameter.cols(
forwardStep, forwardStep + batchStep);
hiddenLayer.cols(forwardStep, forwardStep + batchStep).each_col() +=
input2HiddenBias;
hiddenLayerActivation.cols(forwardStep, forwardStep + batchStep) =
arma::tanh(hiddenLayer.cols(forwardStep, forwardStep + batchStep));
if (forwardStep == 0)
if (!this->HasPreviousStep())
{
cell.cols(forwardStep, forwardStep + batchStep) =
inputGateActivation.cols(forwardStep, forwardStep + batchStep) %
hiddenLayerActivation.cols(forwardStep, forwardStep + batchStep);
cell.slice(this->CurrentStep()) =
inputGateActivation.slice(this->CurrentStep()) %
hiddenLayerActivation.slice(this->CurrentStep());
}
else
{
cell.cols(forwardStep, forwardStep + batchStep) =
forgetGateActivation.cols(forwardStep, forwardStep + batchStep) %
cell.cols(forwardStep - batchSize, forwardStep - batchSize + batchStep)
+ inputGateActivation.cols(forwardStep, forwardStep + batchStep) %
hiddenLayerActivation.cols(forwardStep, forwardStep + batchStep);
cell.slice(this->CurrentStep()) =
forgetGateActivation.slice(this->CurrentStep()) %
cell.slice(this->PreviousStep()) +
inputGateActivation.slice(this->CurrentStep()) %
hiddenLayerActivation.slice(this->CurrentStep());
}
outputGate.cols(forwardStep, forwardStep + batchStep) = input2GateOutputWeight
* input + output2GateOutputWeight * outParameter.cols(
forwardStep, forwardStep + batchStep) + cell.cols(forwardStep,
forwardStep + batchStep).each_col() % cell2GateOutputWeight;
outputGate.cols(forwardStep, forwardStep + batchStep).each_col() +=
input2GateOutputBias;
outputGateActivation.cols(forwardStep, forwardStep + batchStep) = 1.0 /
(1 + arma::exp(-outputGate.cols(forwardStep, forwardStep + batchStep)));
cellActivation.cols(forwardStep, forwardStep + batchStep) =
arma::tanh(cell.cols(forwardStep, forwardStep + batchStep));
outParameter.cols(forwardStep + batchSize,
forwardStep + batchSize + batchStep) =
cellActivation.cols(forwardStep, forwardStep + batchStep) %
outputGateActivation.cols(forwardStep, forwardStep + batchStep);
output = OutputType(outParameter.memptr() +
(forwardStep + batchSize) * outSize, outSize, batchSize, false, false);
cellState = OutputType(cell.memptr() +
forwardStep * outSize, outSize, batchSize, false, false);
forwardStep += batchSize;
if ((forwardStep / batchSize) == bpttSteps)
outputGate = input2GateOutputWeight * input +
cell.slice(this->CurrentStep()).each_col() % cell2GateOutputWeight;
if (this->HasPreviousStep())
{
forwardStep = 0;
outputGate +=
output2GateOutputWeight * outParameter.slice(this->PreviousStep());
}
outputGate.each_col() += input2GateOutputBias;
outputGateActivation.slice(this->CurrentStep()) =
1.0 / (1.0 + arma::exp(-outputGate));
cellActivation.slice(this->CurrentStep()) =
arma::tanh(cell.slice(this->CurrentStep()));
// There's a bit of an issue here: we need to preserve the output for the next
// time step, but we also need to set `output` to that. Unfortunately for now
// we make a copy, but it's possible that we could instead use an alias here,
// or have `outParameter` hold a collection of aliases.
outParameter.slice(this->CurrentStep()) =
cellActivation.slice(this->CurrentStep()) %
outputGateActivation.slice(this->CurrentStep());
output = outParameter.slice(this->CurrentStep());
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename ErrorType, typename GradientType>
void LSTM<InputDataType, OutputDataType>::Backward(
const InputType& /* input */, const ErrorType& gy, GradientType& g)
template<typename MatType>
void LSTMType<MatType>::Backward(
const MatType& /* input */, const MatType& gy, MatType& g)
{
ErrorType gyLocal;
if (gradientStepIdx > 0)
MatType gyLocal;
if (this->HasPreviousStep())
{
gyLocal = gy + prevError;
gyLocal = gy + output2GateOutputWeight.t() * outputGateError +
output2GateForgetWeight.t() * forgetGateError +
output2GateInputWeight.t() * inputGateError +
output2HiddenWeight.t() * hiddenError;
}
else
{
// Make an alias.
gyLocal = ErrorType(((ErrorType&) gy).memptr(), gy.n_rows, gy.n_cols, false,
false);
gyLocal = MatType(((MatType&) gy).memptr(), gy.n_rows, gy.n_cols,
false, false);
}
outputGateError =
gyLocal % cellActivation.cols(backwardStep - batchStep, backwardStep) %
(outputGateActivation.cols(backwardStep - batchStep, backwardStep) %
(1.0 - outputGateActivation.cols(backwardStep - batchStep,
backwardStep)));
outputGateError = gyLocal % cellActivation.slice(this->CurrentStep()) %
(outputGateActivation.slice(this->CurrentStep()) %
(1.0 - outputGateActivation.slice(this->CurrentStep())));
OutputDataType cellError = gyLocal %
outputGateActivation.cols(backwardStep - batchStep, backwardStep) %
(1 - arma::pow(cellActivation.cols(backwardStep -
batchStep, backwardStep), 2)) + outputGateError.each_col() %
cell2GateOutputWeight;
MatType cellError = gyLocal %
outputGateActivation.slice(this->CurrentStep()) %
(1 - arma::pow(cellActivation.slice(this->CurrentStep()), 2)) +
outputGateError.each_col() % cell2GateOutputWeight;
if (gradientStepIdx > 0)
if (this->HasPreviousStep())
{
cellError += inputCellError;
}
if (backwardStep > batchStep)
if (this->HasPreviousStep())
{
forgetGateError = cell.cols((backwardStep - batchSize) - batchStep,
(backwardStep - batchSize)) % cellError % (forgetGateActivation.cols(
backwardStep - batchStep, backwardStep) % (1.0 -
forgetGateActivation.cols(backwardStep - batchStep, backwardStep)));
forgetGateError = cell.slice(this->PreviousStep()) % cellError %
(forgetGateActivation.slice(this->CurrentStep()) %
(1.0 - forgetGateActivation.slice(this->CurrentStep())));
}
else
{
forgetGateError.zeros();
forgetGateError.zeros(forgetGateActivation.n_rows,
forgetGateActivation.n_cols);
}
inputGateError = hiddenLayerActivation.cols(backwardStep - batchStep,
backwardStep) % cellError %
(inputGateActivation.cols(backwardStep - batchStep, backwardStep) %
(1.0 - inputGateActivation.cols(backwardStep - batchStep, backwardStep)));
inputGateError = hiddenLayerActivation.slice(this->CurrentStep()) %
cellError % (inputGateActivation.slice(this->CurrentStep()) %
(1.0 - inputGateActivation.slice(this->CurrentStep())));
hiddenError = inputGateActivation.cols(backwardStep - batchStep,
backwardStep) % cellError % (1 - arma::pow(hiddenLayerActivation.cols(
backwardStep - batchStep, backwardStep), 2));
hiddenError = inputGateActivation.slice(this->CurrentStep()) % cellError %
(1 - arma::pow(hiddenLayerActivation.slice(this->CurrentStep()), 2));
inputCellError = forgetGateActivation.cols(backwardStep - batchStep,
backwardStep) % cellError + forgetGateError.each_col() %
cell2GateForgetWeight + inputGateError.each_col() % cell2GateInputWeight;
inputCellError = forgetGateActivation.slice(this->CurrentStep()) % cellError +
forgetGateError.each_col() % cell2GateForgetWeight +
inputGateError.each_col() % cell2GateInputWeight;
g = input2GateInputWeight.t() * inputGateError +
input2HiddenWeight.t() * hiddenError +
input2GateForgetWeight.t() * forgetGateError +
input2GateOutputWeight.t() * outputGateError;
prevError = output2GateOutputWeight.t() * outputGateError +
output2GateForgetWeight.t() * forgetGateError +
output2GateInputWeight.t() * inputGateError +
output2HiddenWeight.t() * hiddenError;
backwardStep -= batchSize;
gradientStepIdx++;
if (gradientStepIdx == bpttSteps)
{
backwardStep = bpttSteps - 1;
gradientStepIdx = 0;
}
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename ErrorType, typename GradientType>
void LSTM<InputDataType, OutputDataType>::Gradient(
const InputType& input,
const ErrorType& /* error */,
GradientType& gradient)
template<typename MatType>
void LSTMType<MatType>::Gradient(
const MatType& input,
const MatType& /* error */,
MatType& gradient)
{
// This implementation depends on Gradient() being called just after
// Backward(), which is something we can safely assume.
// Input2GateOutputWeight and input2GateOutputBias gradients.
gradient.submat(0, 0, input2GateOutputWeight.n_elem - 1, 0) =
arma::vectorise(outputGateError * input.t());
@@ -450,7 +316,7 @@ void LSTM<InputDataType, OutputDataType>::Gradient(
arma::sum(outputGateError, 1);
size_t offset = input2GateOutputWeight.n_elem + input2GateOutputBias.n_elem;
// Input2GateForgetWeight and input2GateForgetBias gradients.
// input2GateForgetWeight and input2GateForgetBias gradients.
gradient.submat(offset, 0, offset + input2GateForgetWeight.n_elem - 1, 0) =
arma::vectorise(forgetGateError * input.t());
gradient.submat(offset + input2GateForgetWeight.n_elem, 0,
@@ -458,7 +324,7 @@ void LSTM<InputDataType, OutputDataType>::Gradient(
input2GateForgetBias.n_elem - 1, 0) = arma::sum(forgetGateError, 1);
offset += input2GateForgetWeight.n_elem + input2GateForgetBias.n_elem;
// Input2GateInputWeight and input2GateInputBias gradients.
// input2GateInputWeight and input2GateInputBias gradients.
gradient.submat(offset, 0, offset + input2GateInputWeight.n_elem - 1, 0) =
arma::vectorise(inputGateError * input.t());
gradient.submat(offset + input2GateInputWeight.n_elem, 0,
@@ -466,7 +332,7 @@ void LSTM<InputDataType, OutputDataType>::Gradient(
input2GateInputBias.n_elem - 1, 0) = arma::sum(inputGateError, 1);
offset += input2GateInputWeight.n_elem + input2GateInputBias.n_elem;
// Input2HiddenWeight and input2HiddenBias gradients.
// input2HiddenWeight and input2HiddenBias gradients.
gradient.submat(offset, 0, offset + input2HiddenWeight.n_elem - 1, 0) =
arma::vectorise(hiddenError * input.t());
gradient.submat(offset + input2HiddenWeight.n_elem, 0,
@@ -474,48 +340,43 @@ void LSTM<InputDataType, OutputDataType>::Gradient(
arma::sum(hiddenError, 1);
offset += input2HiddenWeight.n_elem + input2HiddenBias.n_elem;
// Output2GateOutputWeight gradients.
// output2GateOutputWeight gradients.
gradient.submat(offset, 0, offset + output2GateOutputWeight.n_elem - 1, 0) =
arma::vectorise(outputGateError *
outParameter.cols(gradientStep - batchStep, gradientStep).t());
outParameter.slice(this->CurrentStep()).t());
offset += output2GateOutputWeight.n_elem;
// Output2GateForgetWeight gradients.
// output2GateForgetWeight gradients.
gradient.submat(offset, 0, offset + output2GateForgetWeight.n_elem - 1, 0) =
arma::vectorise(forgetGateError *
outParameter.cols(gradientStep - batchStep, gradientStep).t());
outParameter.slice(this->CurrentStep()).t());
offset += output2GateForgetWeight.n_elem;
// Output2GateInputWeight gradients.
// output2GateInputWeight gradients.
gradient.submat(offset, 0, offset + output2GateInputWeight.n_elem - 1, 0) =
arma::vectorise(inputGateError *
outParameter.cols(gradientStep - batchStep, gradientStep).t());
outParameter.slice(this->CurrentStep()).t());
offset += output2GateInputWeight.n_elem;
// Output2HiddenWeight gradients.
// output2HiddenWeight gradients.
gradient.submat(offset, 0, offset + output2HiddenWeight.n_elem - 1, 0) =
arma::vectorise(hiddenError *
outParameter.cols(gradientStep - batchStep, gradientStep).t());
outParameter.slice(this->CurrentStep()).t());
offset += output2HiddenWeight.n_elem;
// Cell2GateOutputWeight gradients.
// cell2GateOutputWeight gradients.
gradient.submat(offset, 0, offset + cell2GateOutputWeight.n_elem - 1, 0) =
arma::sum(outputGateError %
cell.cols(gradientStep - batchStep, gradientStep), 1);
arma::sum(outputGateError % cell.slice(this->CurrentStep()), 1);
offset += cell2GateOutputWeight.n_elem;
// Cell2GateForgetWeight and cell2GateInputWeight gradients.
if (gradientStep > batchStep)
// cell2GateForgetWeight and cell2GateInputWeight gradients.
if (this->HasPreviousStep())
{
gradient.submat(offset, 0, offset + cell2GateForgetWeight.n_elem - 1, 0) =
arma::sum(forgetGateError %
cell.cols((gradientStep - batchSize) - batchStep,
(gradientStep - batchSize)), 1);
arma::sum(forgetGateError % cell.slice(this->PreviousStep()), 1);
gradient.submat(offset + cell2GateForgetWeight.n_elem, 0, offset +
cell2GateForgetWeight.n_elem + cell2GateInputWeight.n_elem - 1, 0) =
arma::sum(inputGateError %
cell.cols((gradientStep - batchSize) - batchStep,
(gradientStep - batchSize)), 1);
arma::sum(inputGateError % cell.slice(this->PreviousStep()), 1);
}
else
{
@@ -525,41 +386,32 @@ void LSTM<InputDataType, OutputDataType>::Gradient(
cell2GateForgetWeight.n_elem +
cell2GateInputWeight.n_elem - 1, 0).zeros();
}
if (gradientStep == 0)
{
gradientStep = batchSize * bpttSteps - 1;
}
else
{
gradientStep -= batchSize;
}
}
template<typename InputDataType, typename OutputDataType>
template<typename MatType>
template<typename Archive>
void LSTM<InputDataType, OutputDataType>::serialize(
Archive& ar, const uint32_t /* version */)
void LSTMType<MatType>::serialize(Archive& ar, const uint32_t /* version */)
{
ar(CEREAL_NVP(weights));
ar(cereal::base_class<RecurrentLayer<MatType>>(this));
ar(CEREAL_NVP(inSize));
ar(CEREAL_NVP(outSize));
ar(CEREAL_NVP(rho));
ar(CEREAL_NVP(bpttSteps));
ar(CEREAL_NVP(batchSize));
ar(CEREAL_NVP(batchStep));
ar(CEREAL_NVP(forwardStep));
ar(CEREAL_NVP(backwardStep));
ar(CEREAL_NVP(gradientStep));
ar(CEREAL_NVP(gradientStepIdx));
ar(CEREAL_NVP(cell));
ar(CEREAL_NVP(inputGateActivation));
ar(CEREAL_NVP(forgetGateActivation));
ar(CEREAL_NVP(outputGateActivation));
ar(CEREAL_NVP(hiddenLayerActivation));
ar(CEREAL_NVP(cellActivation));
ar(CEREAL_NVP(prevError));
ar(CEREAL_NVP(outParameter));
// Clear recurrent state if we are loading.
if (Archive::is_loading::value)
{
inputGateActivation.clear();
forgetGateActivation.clear();
outputGateActivation.clear();
hiddenLayerActivation.clear();
cellActivation.clear();
forgetGateError.clear();
outputGateError.clear();
outParameter.clear();
inputCellError.clear();
inputGateError.clear();
hiddenError.clear();
}
}
} // namespace ann
src/mlpack/methods/ann/layer/max_pooling.hpp
+128 -158
View File
@@ -15,6 +15,8 @@
#include <mlpack/prereqs.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -28,32 +30,39 @@ class MaxPoolingRule
/*
* Return the maximum value within the receptive block.
*
* @param input Input used to perform the pooling operation.
* @param input Input used to perform the pooling operation. Could be an
* Armadillo subview.
*/
template<typename MatType>
size_t Pooling(const MatType& input)
typename MatType::elem_type Pooling(const MatType& input)
{
return arma::as_scalar(arma::find(input.max() == input, 1));
return arma::max(arma::vectorise(input));
}
template<typename MatType>
std::tuple<size_t, typename MatType::elem_type> PoolingWithIndex(
const MatType& input)
{
const typename MatType::elem_type maxVal =
arma::max(arma::vectorise(input));
const size_t index = arma::as_scalar(arma::find(input == maxVal, 1));
return std::tuple<size_t, typename MatType::elem_type>(index, maxVal);
}
};
/**
* Implementation of the MaxPooling layer.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class MaxPooling
template<typename MatType = arma::mat>
class MaxPoolingType : public Layer<MatType>
{
public:
//! Create the MaxPooling object.
MaxPooling();
MaxPoolingType();
/**
* Create the MaxPooling object using the specified number of units.
@@ -64,11 +73,25 @@ class MaxPooling
* @param strideHeight Width of the stride operation.
* @param floor Rounding operator (floor or ceil).
*/
MaxPooling(const size_t kernelWidth,
const size_t kernelHeight,
const size_t strideWidth = 1,
const size_t strideHeight = 1,
const bool floor = true);
MaxPoolingType(const size_t kernelWidth,
const size_t kernelHeight,
const size_t strideWidth = 1,
const size_t strideHeight = 1,
const bool floor = true);
// Virtual destructor.
virtual ~MaxPoolingType() { }
//! Copy the given MaxPoolingType.
MaxPoolingType(const MaxPoolingType& other);
//! Take ownership of the given MaxPoolingType.
MaxPoolingType(MaxPoolingType&& other);
//! Copy the given MaxPoolingType.
MaxPoolingType& operator=(const MaxPoolingType& other);
//! Take ownership of the given MaxPoolingType.
MaxPoolingType& operator=(MaxPoolingType&& other);
MaxPoolingType* Clone() const { return new MaxPoolingType(*this); }
/**
* Ordinary feed forward pass of a neural network, evaluating the function
@@ -77,8 +100,7 @@ class MaxPooling
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of a neural network, using 3rd-order tensors as
@@ -89,79 +111,37 @@ class MaxPooling
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
//! Get the output parameter.
const OutputDataType& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
const OutputDataType& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the input width.
size_t InputWidth() const { return inputWidth; }
//! Modify the input width.
size_t& InputWidth() { return inputWidth; }
//! Get the input height.
size_t InputHeight() const { return inputHeight; }
//! Modify the input height.
size_t& InputHeight() { return inputHeight; }
//! Get the output width.
size_t OutputWidth() const { return outputWidth; }
//! Modify the output width.
size_t& OutputWidth() { return outputWidth; }
//! Get the output height.
size_t OutputHeight() const { return outputHeight; }
//! Modify the output height.
size_t& OutputHeight() { return outputHeight; }
//! Get the input size.
size_t InputSize() const { return inSize; }
//! Get the output size.
size_t OutputSize() const { return outSize; }
void Backward(const MatType& /* input */,
const MatType& gy,
MatType& g);
//! Get the kernel width.
size_t KernelWidth() const { return kernelWidth; }
size_t const& KernelWidth() const { return kernelWidth; }
//! Modify the kernel width.
size_t& KernelWidth() { return kernelWidth; }
//! Get the kernel height.
size_t KernelHeight() const { return kernelHeight; }
size_t const& KernelHeight() const { return kernelHeight; }
//! Modify the kernel height.
size_t& KernelHeight() { return kernelHeight; }
//! Get the stride width.
size_t StrideWidth() const { return strideWidth; }
size_t const& StrideWidth() const { return strideWidth; }
//! Modify the stride width.
size_t& StrideWidth() { return strideWidth; }
//! Get the stride height.
size_t StrideHeight() const { return strideHeight; }
size_t const& StrideHeight() const { return strideHeight; }
//! Modify the stride height.
size_t& StrideHeight() { return strideHeight; }
//! Get the value of the rounding operation.
bool Floor() const { return floor; }
bool const& Floor() const { return floor; }
//! Modify the value of the rounding operation.
bool& Floor() { return floor; }
//! Get the value of the deterministic parameter.
bool Deterministic() const { return deterministic; }
//! Modify the value of the deterministic parameter.
bool& Deterministic() { return deterministic; }
//! Get the size of the weights.
size_t WeightSize() const { return 0; }
//! Compute the size of the output given `InputDimensions()`.
void ComputeOutputDimensions();
/**
* Serialize the layer.
@@ -170,62 +150,94 @@ class MaxPooling
void serialize(Archive& ar, const uint32_t /* version */);
private:
/**
/**
* Apply pooling to the input and store the results.
*
* @param input The input to be apply the pooling rule.
* @param output The pooled result.
* @param poolingIndices The pooled indices.
*/
template<typename eT>
void PoolingOperation(const arma::Mat<eT>& input,
arma::Mat<eT>& output,
arma::Mat<eT>& poolingIndices)
void PoolingOperation(
const arma::Cube<typename MatType::elem_type>& input,
arma::Cube<typename MatType::elem_type>& output,
arma::Cube<size_t>& poolingIndices)
{
for (size_t j = 0, colidx = 0; j < output.n_cols;
++j, colidx += strideHeight)
// Iterate over all slices individually.
for (size_t s = 0; s < input.n_slices; ++s)
{
for (size_t i = 0, rowidx = 0; i < output.n_rows;
++i, rowidx += strideWidth)
for (size_t j = 0, colidx = 0; j < output.n_cols;
++j, colidx += strideHeight)
{
size_t rowEnd = rowidx + kernelWidth - 1;
size_t colEnd = colidx + kernelHeight - 1;
if (rowEnd > input.n_rows - 1)
rowEnd = input.n_rows - 1;
if (colEnd > input.n_cols - 1)
colEnd = input.n_cols - 1;
arma::mat subInput = input(
arma::span(rowidx, rowEnd),
arma::span(colidx, colEnd));
const size_t idx = pooling.Pooling(subInput);
output(i, j) = subInput(idx);
if (!deterministic)
for (size_t i = 0, rowidx = 0; i < output.n_rows;
++i, rowidx += strideWidth)
{
arma::Mat<size_t> subIndices = indices(arma::span(rowidx, rowEnd),
arma::span(colidx, colEnd));
const std::tuple<size_t, typename MatType::elem_type> poolResult =
pooling.PoolingWithIndex(input.slice(s).submat(
rowidx,
colidx,
rowidx + kernelWidth - 1 - offset,
colidx + kernelHeight - 1 - offset));
poolingIndices(i, j) = subIndices(idx);
// Now map the returned pooling index, which corresponds to the
// submatrix we gave, back to its position in the (linearized) input.
const size_t poolIndex = std::get<0>(poolResult);
const size_t poolingCol = poolIndex / (kernelWidth - offset);
const size_t poolingRow = poolIndex % (kernelWidth - offset);
const size_t unmappedPoolingIndex = (rowidx + poolingRow) +
input.n_rows * (colidx + poolingCol) +
input.n_rows * input.n_cols * s;
poolingIndices(i, j, s) = unmappedPoolingIndex;
output(i, j, s) = std::get<1>(poolResult);
}
}
}
}
/**
* Apply unpooling to the input and store the results.
* Apply pooling to all slices of the input and store the results, but not the
* indices used.
*
* @param input The input to apply the pooling rule to.
* @param output The pooled result.
*/
void PoolingOperation(
const arma::Cube<typename MatType::elem_type>& input,
arma::Cube<typename MatType::elem_type>& output)
{
// Iterate over all slices individually.
for (size_t s = 0; s < input.n_slices; ++s)
{
for (size_t j = 0, colidx = 0; j < output.n_cols;
++j, colidx += strideHeight)
{
for (size_t i = 0, rowidx = 0; i < output.n_rows;
++i, rowidx += strideWidth)
{
output(i, j, s) = pooling.Pooling(input.slice(s).submat(
rowidx,
colidx,
rowidx + kernelWidth - 1 - offset,
colidx + kernelHeight - 1 - offset));
}
}
}
}
/**
* Apply unpooling to all slices of the input and store the results.
*
* @param error The backward error.
* @param output The pooled result.
* @param poolingIndices The pooled indices.
* @param poolingIndices The pooled indices (from `PoolingOperation()`).
*/
template<typename eT>
void Unpooling(const arma::Mat<eT>& error,
arma::Mat<eT>& output,
arma::Mat<eT>& poolingIndices)
void UnpoolingOperation(
const arma::Cube<typename MatType::elem_type>& error,
arma::Cube<typename MatType::elem_type>& output,
const arma::Cube<size_t>& poolingIndices)
{
output.zeros();
for (size_t i = 0; i < poolingIndices.n_elem; ++i)
{
output(poolingIndices(i)) += error(i);
@@ -247,64 +259,22 @@ class MaxPooling
//! Rounding operation used.
bool floor;
//! Locally-stored number of input channels.
size_t inSize;
//! Locally-stored number of channels.
size_t channels;
//! Locally-stored number of output channels.
size_t outSize;
//! Locally-stored reset parameter used to initialize the module once.
bool reset;
//! Locally-stored input width.
size_t inputWidth;
//! Locally-stored input height.
size_t inputHeight;
//! Locally-stored output width.
size_t outputWidth;
//! Locally-stored output height.
size_t outputHeight;
//! If true use maximum a posteriori during the forward pass.
bool deterministic;
//! Locally-stored number of input units.
size_t batchSize;
//! Locally-stored output parameter.
arma::cube outputTemp;
//! Locally-stored transformed input parameter.
arma::cube inputTemp;
//! Locally-stored transformed output parameter.
arma::cube gTemp;
//! Locally-stored offset: indicates whether we take the first element or the
//! second element when pooling. Computed by `ComputeOutputDimensions()`.
size_t offset;
//! Locally-stored pooling strategy.
MaxPoolingRule pooling;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored pooling indices.
arma::Cube<size_t> poolingIndices;
}; // class MaxPoolingType
//! Locally-stored gradient object.
OutputDataType gradient;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
//! Locally-stored indices matrix parameter.
arma::Mat<size_t> indices;
//! Locally-stored indices column parameter.
arma::Col<size_t> indicesCol;
//! Locally-stored pooling indicies.
std::vector<arma::cube> poolingIndices;
}; // class MaxPooling
// Standard MaxPooling layer.
typedef MaxPoolingType<arma::mat> MaxPooling;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/max_pooling_impl.hpp
+154 -94
View File
@@ -19,141 +19,201 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
MaxPooling<InputDataType, OutputDataType>::MaxPooling()
template<typename MatType>
MaxPoolingType<MatType>::MaxPoolingType() :
Layer<MatType>()
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
MaxPooling<InputDataType, OutputDataType>::MaxPooling(
template<typename MatType>
MaxPoolingType<MatType>::MaxPoolingType(
const size_t kernelWidth,
const size_t kernelHeight,
const size_t strideWidth,
const size_t strideHeight,
const bool floor) :
Layer<MatType>(),
kernelWidth(kernelWidth),
kernelHeight(kernelHeight),
strideWidth(strideWidth),
strideHeight(strideHeight),
floor(floor),
inSize(0),
outSize(0),
reset(false),
inputWidth(0),
inputHeight(0),
outputWidth(0),
outputHeight(0),
deterministic(false),
batchSize(0)
channels(0),
offset(0)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void MaxPooling<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
template<typename MatType>
MaxPoolingType<MatType>::MaxPoolingType(
const MaxPoolingType& other) :
Layer<MatType>(other),
kernelWidth(other.kernelWidth),
kernelHeight(other.kernelHeight),
strideWidth(other.strideWidth),
strideHeight(other.strideHeight),
floor(other.floor),
channels(other.channels),
offset(other.offset),
pooling(other.pooling)
{
batchSize = input.n_cols;
inSize = input.n_elem / (inputWidth * inputHeight * batchSize);
inputTemp = arma::cube(const_cast<arma::Mat<eT>&>(input).memptr(),
inputWidth, inputHeight, batchSize * inSize, false, false);
// Nothing to do here.
}
if (floor)
template<typename MatType>
MaxPoolingType<MatType>::MaxPoolingType(
MaxPoolingType&& other) :
Layer<MatType>(std::move(other)),
kernelWidth(std::move(other.kernelWidth)),
kernelHeight(std::move(other.kernelHeight)),
strideWidth(std::move(other.strideWidth)),
strideHeight(std::move(other.strideHeight)),
floor(std::move(other.floor)),
channels(std::move(other.channels)),
offset(std::move(other.offset)),
pooling(std::move(other.pooling))
{
// Nothing to do here.
}
template<typename MatType>
MaxPoolingType<MatType>&
MaxPoolingType<MatType>::operator=(const MaxPoolingType& other)
{
if (&other != this)
{
outputWidth = std::floor((inputWidth -
(double) kernelWidth) / (double) strideWidth + 1);
outputHeight = std::floor((inputHeight -
(double) kernelHeight) / (double) strideHeight + 1);
Layer<MatType>::operator=(other);
kernelWidth = other.kernelWidth;
kernelHeight = other.kernelHeight;
strideWidth = other.strideWidth;
strideHeight = other.strideHeight;
floor = other.floor;
channels = other.channels;
offset = other.offset;
pooling = other.pooling;
}
return *this;
}
template<typename MatType>
MaxPoolingType<MatType>&
MaxPoolingType<MatType>::operator=(MaxPoolingType&& other)
{
if (&other != this)
{
Layer<MatType>::operator=(std::move(other));
kernelWidth = std::move(other.kernelWidth);
kernelHeight = std::move(other.kernelHeight);
strideWidth = std::move(other.strideWidth);
strideHeight = std::move(other.strideHeight);
floor = std::move(other.floor);
channels = std::move(other.channels);
offset = std::move(other.offset);
pooling = std::move(other.pooling);
}
return *this;
}
template<typename MatType>
void MaxPoolingType<MatType>::Forward(const MatType& input, MatType& output)
{
arma::Cube<typename MatType::elem_type> inputTemp(
const_cast<MatType&>(input).memptr(), this->inputDimensions[0],
this->inputDimensions[1], input.n_cols * channels, false, false);
arma::Cube<typename MatType::elem_type> outputTemp(output.memptr(),
this->outputDimensions[0], this->outputDimensions[1],
input.n_cols * channels, false, true);
if (this->training)
{
// If we are training, we'll do a backwards pass, so we need to ensure that
// we know what indices we used.
poolingIndices.set_size(this->outputDimensions[0],
this->outputDimensions[1], input.n_cols * channels);
PoolingOperation(inputTemp, outputTemp, poolingIndices);
}
else
{
outputWidth = std::ceil((inputWidth -
(double) kernelWidth) / (double) strideWidth + 1);
outputHeight = std::ceil((inputHeight -
(double) kernelHeight) / (double) strideHeight + 1);
PoolingOperation(inputTemp, outputTemp);
}
outputTemp = arma::zeros<arma::Cube<eT> >(outputWidth, outputHeight,
batchSize * inSize);
if (!deterministic)
{
poolingIndices.push_back(outputTemp);
}
if (!reset)
{
size_t elements = inputWidth * inputHeight;
indicesCol = arma::linspace<arma::Col<size_t> >(0, (elements - 1),
elements);
indices = arma::Mat<size_t>(indicesCol.memptr(), inputWidth, inputHeight);
reset = true;
}
for (size_t s = 0; s < inputTemp.n_slices; s++)
{
if (!deterministic)
{
PoolingOperation(inputTemp.slice(s), outputTemp.slice(s),
poolingIndices.back().slice(s));
}
else
{
PoolingOperation(inputTemp.slice(s), outputTemp.slice(s),
inputTemp.slice(s));
}
}
output = arma::Mat<eT>(outputTemp.memptr(), outputTemp.n_elem / batchSize,
batchSize);
outputWidth = outputTemp.n_rows;
outputHeight = outputTemp.n_cols;
outSize = batchSize * inSize;
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void MaxPooling<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& /* input */, const arma::Mat<eT>& gy, arma::Mat<eT>& g)
template<typename MatType>
void MaxPoolingType<MatType>::Backward(
const MatType& input, const MatType& gy, MatType& g)
{
arma::cube mappedError = arma::cube(((arma::Mat<eT>&) gy).memptr(),
outputWidth, outputHeight, outSize, false, false);
arma::Cube<typename MatType::elem_type> mappedError =
arma::Cube<typename MatType::elem_type>(((MatType&) gy).memptr(),
this->outputDimensions[0], this->outputDimensions[1],
channels * input.n_cols, false, false);
gTemp = arma::zeros<arma::cube>(inputTemp.n_rows,
inputTemp.n_cols, inputTemp.n_slices);
arma::Cube<typename MatType::elem_type> gTemp(g.memptr(),
this->inputDimensions[0], this->inputDimensions[1],
channels * input.n_cols, false, true);
for (size_t s = 0; s < mappedError.n_slices; s++)
{
Unpooling(mappedError.slice(s), gTemp.slice(s),
poolingIndices.back().slice(s));
}
poolingIndices.pop_back();
g = arma::mat(gTemp.memptr(), gTemp.n_elem / batchSize, batchSize);
// There's no version of UnpoolingOperation without pooling indices, because
// if we call `Backward()`, we know for sure we are training.
UnpoolingOperation(mappedError, gTemp, poolingIndices);
}
template<typename InputDataType, typename OutputDataType>
template<typename MatType>
void MaxPoolingType<MatType>::ComputeOutputDimensions()
{
this->outputDimensions = this->inputDimensions;
// Compute the size of the output.
if (floor)
{
this->outputDimensions[0] = std::floor((this->inputDimensions[0] -
(double) kernelWidth) / (double) strideWidth + 1);
this->outputDimensions[1] = std::floor((this->inputDimensions[1] -
(double) kernelHeight) / (double) strideHeight + 1);
offset = 0;
}
else
{
this->outputDimensions[0] = std::ceil((this->inputDimensions[0] -
(double) kernelWidth) / (double) strideWidth + 1);
this->outputDimensions[1] = std::ceil((this->inputDimensions[1] -
(double) kernelHeight) / (double) strideHeight + 1);
offset = 1;
}
// Higher dimensions are not modified.
// Cache input size and output size.
channels = 1;
for (size_t i = 2; i < this->inputDimensions.size(); ++i)
channels *= this->inputDimensions[i];
}
template<typename MatType>
template<typename Archive>
void MaxPooling<InputDataType, OutputDataType>::serialize(
void MaxPoolingType<MatType>::serialize(
Archive& ar,
const uint32_t /* version */)
{
ar(cereal::base_class<Layer<MatType>>(this));
ar(CEREAL_NVP(kernelWidth));
ar(CEREAL_NVP(kernelHeight));
ar(CEREAL_NVP(strideWidth));
ar(CEREAL_NVP(strideHeight));
ar(CEREAL_NVP(batchSize));
ar(CEREAL_NVP(channels));
ar(CEREAL_NVP(floor));
ar(CEREAL_NVP(inputWidth));
ar(CEREAL_NVP(inputHeight));
ar(CEREAL_NVP(outputWidth));
ar(CEREAL_NVP(outputHeight));
ar(CEREAL_NVP(offset));
if (Archive::is_loading::value)
{
// Clear any memory used by `poolingIndices`.
poolingIndices.clear();
}
}
} // namespace ann
src/mlpack/methods/ann/layer/mean_pooling_impl.hpp
-134
View File
@@ -1,134 +0,0 @@
/**
* @file methods/ann/layer/mean_pooling_impl.hpp
* @author Marcus Edel
* @author Nilay Jain
*
* Implementation of the MeanPooling layer class.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_MEAN_POOLING_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_MEAN_POOLING_IMPL_HPP
// In case it hasn't yet been included.
#include "mean_pooling.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
MeanPooling<InputDataType, OutputDataType>::MeanPooling()
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
MeanPooling<InputDataType, OutputDataType>::MeanPooling(
const size_t kernelWidth,
const size_t kernelHeight,
const size_t strideWidth,
const size_t strideHeight,
const bool floor) :
kernelWidth(kernelWidth),
kernelHeight(kernelHeight),
strideWidth(strideWidth),
strideHeight(strideHeight),
floor(floor),
inSize(0),
outSize(0),
inputWidth(0),
inputHeight(0),
outputWidth(0),
outputHeight(0),
reset(false),
deterministic(false),
batchSize(0)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void MeanPooling<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
{
batchSize = input.n_cols;
inSize = input.n_elem / (inputWidth * inputHeight * batchSize);
inputTemp = arma::cube(const_cast<arma::Mat<eT>&>(input).memptr(),
inputWidth, inputHeight, batchSize * inSize, false, false);
if (floor)
{
outputWidth = std::floor((inputWidth -
(double) kernelWidth) / (double) strideWidth + 1);
outputHeight = std::floor((inputHeight -
(double) kernelHeight) / (double) strideHeight + 1);
}
else
{
outputWidth = std::ceil((inputWidth -
(double) kernelWidth) / (double) strideWidth + 1);
outputHeight = std::ceil((inputHeight -
(double) kernelHeight) / (double) strideHeight + 1);
}
outputTemp = arma::zeros<arma::Cube<eT> >(outputWidth, outputHeight,
batchSize * inSize);
for (size_t s = 0; s < inputTemp.n_slices; s++)
Pooling(inputTemp.slice(s), outputTemp.slice(s));
output = arma::Mat<eT>(outputTemp.memptr(), outputTemp.n_elem / batchSize,
batchSize);
outputWidth = outputTemp.n_rows;
outputHeight = outputTemp.n_cols;
outSize = batchSize * inSize;
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void MeanPooling<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g)
{
arma::cube mappedError = arma::cube(((arma::Mat<eT>&) gy).memptr(),
outputWidth, outputHeight, outSize, false, false);
gTemp = arma::zeros<arma::cube>(inputTemp.n_rows,
inputTemp.n_cols, inputTemp.n_slices);
for (size_t s = 0; s < mappedError.n_slices; s++)
{
Unpooling(inputTemp.slice(s), mappedError.slice(s), gTemp.slice(s));
}
g = arma::mat(gTemp.memptr(), gTemp.n_elem / batchSize, batchSize);
}
template<typename InputDataType, typename OutputDataType>
template<typename Archive>
void MeanPooling<InputDataType, OutputDataType>::serialize(
Archive& ar,
const uint32_t /* version */)
{
ar(CEREAL_NVP(kernelWidth));
ar(CEREAL_NVP(kernelHeight));
ar(CEREAL_NVP(strideWidth));
ar(CEREAL_NVP(strideHeight));
ar(CEREAL_NVP(batchSize));
ar(CEREAL_NVP(floor));
ar(CEREAL_NVP(inputWidth));
ar(CEREAL_NVP(inputHeight));
ar(CEREAL_NVP(outputWidth));
ar(CEREAL_NVP(outputHeight));
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/minibatch_discrimination_impl.hpp
-147
View File
@@ -1,147 +0,0 @@
/**
* @file methods/ann/layer/minibatch_discrimination_impl.hpp
* @author Saksham Bansal
*
* Implementation of the MiniBatchDiscrimination layer class.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_MINIBATCH_DISCRIMINATION_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_MINIBATCH_DISCRIMINATION_IMPL_HPP
// In case it hasn't yet been included.
#include "minibatch_discrimination.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
MiniBatchDiscrimination<InputDataType, OutputDataType
>::MiniBatchDiscrimination() :
A(0),
B(0),
C(0),
batchSize(0)
{
// Nothing to do here.
}
template <typename InputDataType, typename OutputDataType>
MiniBatchDiscrimination<InputDataType, OutputDataType
>::MiniBatchDiscrimination(
const size_t inSize,
const size_t outSize,
const size_t features) :
A(inSize),
B(outSize - inSize),
C(features),
batchSize(0)
{
weights.set_size(A * B * C, 1);
}
template<typename InputDataType, typename OutputDataType>
void MiniBatchDiscrimination<InputDataType, OutputDataType>::Reset()
{
weight = arma::mat(weights.memptr(), B * C, A, false, false);
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void MiniBatchDiscrimination<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
{
batchSize = input.n_cols;
tempM = weight * input;
M = arma::cube(tempM.memptr(), B, C, batchSize, false, false);
distances.set_size(B, batchSize, batchSize);
output.set_size(B, batchSize);
for (size_t i = 0; i < M.n_slices; ++i)
{
output.col(i).ones();
for (size_t j = 0; j < M.n_slices; ++j)
{
if (j < i)
{
output.col(i) += distances.slice(j).col(i);
}
else if (i == j)
{
continue;
}
else
{
distances.slice(i).col(j) =
arma::exp(-arma::sum(abs(M.slice(i) - M.slice(j)), 1));
output.col(i) += distances.slice(i).col(j);
}
}
}
output = join_cols(input, output); // (A + B) x batchSize
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void MiniBatchDiscrimination<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& /* input */, const arma::Mat<eT>& gy, arma::Mat<eT>& g)
{
g = gy.head_rows(A);
arma::Mat<eT> gM = gy.tail_rows(B);
deltaM.zeros(B, C, batchSize);
for (size_t i = 0; i < M.n_slices; ++i)
{
for (size_t j = 0; j < M.n_slices; ++j)
{
if (i == j)
{
continue;
}
arma::mat t = arma::sign(M.slice(i) - M.slice(j));
t.each_col() %=
distances.slice(std::min(i, j)).col(std::max(i, j)) % gM.col(i);
deltaM.slice(i) -= t;
deltaM.slice(j) += t;
}
}
deltaTemp = arma::mat(deltaM.memptr(), B * C, batchSize, false, false);
g += weight.t() * deltaTemp;
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void MiniBatchDiscrimination<InputDataType, OutputDataType>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& /* error */,
arma::Mat<eT>& gradient)
{
gradient = arma::vectorise(deltaTemp * input.t());
}
template<typename InputDataType, typename OutputDataType>
template<typename Archive>
void MiniBatchDiscrimination<InputDataType, OutputDataType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(CEREAL_NVP(A));
ar(CEREAL_NVP(B));
ar(CEREAL_NVP(C));
// This is inefficient, but we have to allocate this memory so that
// WeightSetVisitor gets the right size.
if (cereal::is_loading<Archive>())
{
weights.set_size(A * B * C, 1);
}
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/multi_layer.hpp
+252
View File
@@ -0,0 +1,252 @@
/**
* @file methods/ann/layer/multi_layer.hpp
* @author Ryan Curtin
*
* Base class for neural network layers that are wrappers around other layers.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_MULTI_LAYER_HPP
#define MLPACK_METHODS_ANN_LAYER_MULTI_LAYER_HPP
#include "../make_alias.hpp"
#include "layer.hpp"
namespace mlpack {
namespace ann {
/**
* A "multi-layer" is a layer that is a wrapper around other layers. It passes
* the input through all of its child layers sequentially, returning the output
* from the last layer.
*
* It's likely not very useful to use this layer directly; instead, this layer
* is meant as a base class for use by other layers that must store and use
* multiple layers.
*
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template<typename MatType>
class MultiLayer : public Layer<MatType>
{
public:
/**
* Create an empty MultiLayer that holds no layers of its own. Be sure to add
* layers with Add() before using!
*/
MultiLayer();
//! Copy the given MultiLayer.
MultiLayer(const MultiLayer& other);
//! Take ownership of the layers of the given MultiLayer.
MultiLayer(MultiLayer&& other);
//! Copy the given MultiLayer.
MultiLayer& operator=(const MultiLayer& other);
//! Take ownership of the given MultiLayer.
MultiLayer& operator=(MultiLayer&& other);
//! Virtual destructor: delete all held layers.
virtual ~MultiLayer()
{
for (size_t i = 0; i < network.size(); ++i)
delete network[i];
}
//! Create a copy of the MultiLayer (this is safe for polymorphic use).
virtual MultiLayer* Clone() const { return new MultiLayer(*this); }
/**
* Perform a forward pass with the given input data. `output` is expected to
* have the correct size (e.g. number of rows equal to `OutputSize()` of the
* last held layer; number of columns equal to `input.n_cols`).
*
* @param input Input data to pass through the MultiLayer.
* @param output Matrix to store output in.
*/
virtual void Forward(const MatType& input, MatType& output);
/**
* Perform a forward pass with the given input data, but only on a subset of
* the layers in the MultiLayer. `output` is expected to have the correct
* size (e.g. number of rows equal to `OutputSize()` of the last layer to be
* computed; number of columns equal to `input.n_cols`).
*
* @param input Input data to pass through the MultiLayer.
* @param output Matrix to store output in.
* @param start Index of first layer to pass data through.
* @param end Index of last layer to pass data through.
*/
void Forward(const MatType& input,
MatType& output,
const size_t start,
const size_t end);
/**
* Perform a backward pass with the given data. `gy` is expected to be the
* propagated error from the subsequent layer (or output), `input` is expected
* to be the output from this layer when `Forward()` was called, and `g` will
* store the propagated error from this layer (to be passed to the previous
* layer as `gy`).
*
* It is expected that `g` has the correct size already (e.g., number of rows
* equal to `OutputSize()` of the previous layer, and number of columns equal
* to `input.n_cols`).
*
* This function is expected to be called for the same input data as
* `Forward()` was just called for.
*
* @param input Output of Forward().
* @param gy Propagated error from next layer.
* @param g Matrix to store propagated error in for previous layer.
*/
virtual void Backward(const MatType& input,
const MatType& gy,
MatType& g);
/**
* Compute the gradients of each layer.
*
* This function is expected to be called for the same input data as
* `Forward()` and `Backward()` were just called for. That is, `input` here
* should be the same data as `Forward()` was called with.
*
* `gradient` is expected to have the correct size already (e.g., number of
* rows equal to 1, and number of columns equal to `WeightSize()`).
*
* @param input Original input data provided to Forward().
* @param error Error as computed by `Backward()`.
* @param gradient Matrix to store the gradients in.
*/
virtual void Gradient(const MatType& input,
const MatType& error,
MatType& gradient);
/**
* Set the weights of the layer to use the memory given as `weightsPtr`.
*/
virtual void SetWeights(typename MatType::elem_type* weightsPtr);
/**
* Return the number of weights in the MultiLayer. This is the sum of the
* number of weights in each layer.
*/
virtual size_t WeightSize() const;
/**
* Compute the output dimensions of the MultiLayer using `InputDimensions()`.
* This computes the dimensions of each layer held by the MultiLayer, and the
* output dimensions are set to the output dimensions of the last layer.
*/
virtual void ComputeOutputDimensions();
/**
* Compute the loss that should be added to the objective.
*/
virtual double Loss() const;
/*
* Add a new module to the model.
*
* @param args The layer parameter.
*/
template <typename LayerType, typename... Args>
void Add(Args... args)
{
network.push_back(new LayerType(args...));
layerOutputs.push_back(MatType());
layerDeltas.push_back(MatType());
layerGradients.push_back(MatType());
}
/*
* Add a new module to the model.
*
* @param layer The Layer to be added to the model.
*/
void Add(Layer<MatType>* layer)
{
network.push_back(layer);
layerOutputs.push_back(MatType());
layerDeltas.push_back(MatType());
layerGradients.push_back(MatType());
}
//! Get the network (series of layers) held by this MultiLayer.
const std::vector<Layer<MatType>*> Network() const
{
return network;
}
//! Modify the network (series of layers) held by this MultiLayer. Be
//! careful!
std::vector<Layer<MatType>*>& Network() { return network; }
//! Serialize the MultiLayer.
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
protected:
/**
* Initialize memory that will be used by each layer for the forward pass,
* assuming that the input will have the given `batchSize`. When `Forward()`
* is called, each internally-held layer will output its results into the
* memory allocated by this function (this is the internal member
* `layerOutputMatrix` and its aliases `layerOutputs`).
*/
void InitializeForwardPassMemory(const size_t batchSize);
/**
* Initialize memory that will be used by each layer for the backwards pass,
* assuming that the input will have the given `batchSize`. When `Backward()`
* is called, each internally-held layer will output the results of its
* backwards pass into the memory allocated by this function (this is the
* internal member `layerDeltaMatrix` and its aliases `layerDeltas`).
*/
void InitializeBackwardPassMemory(const size_t batchSize);
/**
* Initialize memory for the gradient pass. This sets the internal aliases
* `layerGradients` appropriately using the memory from the given `gradient`,
* such that each layer will output its gradient (via its `Gradient()` method)
* into the appropriate member of `layerGradients`.
*/
void InitializeGradientPassMemory(MatType& gradient);
//! The internally-held network.
std::vector<Layer<MatType>*> network;
// Total number of elements in the input, cached for convenience.
size_t inSize;
// Total number of input elements for *every* layer.
size_t totalInputSize;
// Total number of output elements for *every* layer.
size_t totalOutputSize;
//! This matrix stores all of the outputs of each layer when Forward() is
//! called. See `InitializeForwardPassMemory()`.
MatType layerOutputMatrix;
//! These are aliases of `layerOutputMatrix` for each layer.
std::vector<MatType> layerOutputs;
//! This matrix stores all of the backwards pass results of each layer when
//! Backward() is called. See `InitializeBackwardPassMemory()`.
MatType layerDeltaMatrix;
//! These are aliases of `layerDeltaMatrix` for each layer.
std::vector<MatType> layerDeltas;
//! Gradient aliases for each layer. Note that this is *only* valid in the
//! context of `Gradient()`! We have it as a class member to avoid
//! reallocating the `MatType`s each call to `Gradient()`.
std::vector<MatType> layerGradients;
};
} // namespace ann
} // namespace mlpack
// Include implementation.
#include "multi_layer_impl.hpp"
#endif
src/mlpack/methods/ann/layer/multi_layer_impl.hpp
+414
View File
@@ -0,0 +1,414 @@
/**
* @file methods/ann/layer/multi_layer_impl.hpp
* @author Ryan Curtin
*
* Implementation of the base class for neural network layers that are wrappers
* around other layers.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_MULTI_LAYER_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_MULTI_LAYER_IMPL_HPP
#include "multi_layer.hpp"
namespace mlpack {
namespace ann {
template<typename MatType>
MultiLayer<MatType>::MultiLayer() :
inSize(0),
totalInputSize(0),
totalOutputSize(0)
{
// Nothing to do.
}
template<typename MatType>
MultiLayer<MatType>::MultiLayer(const MultiLayer& other) :
Layer<MatType>(other),
inSize(other.inSize),
totalInputSize(other.totalInputSize),
totalOutputSize(other.totalOutputSize),
layerOutputMatrix(other.layerOutputMatrix),
layerDeltaMatrix(other.layerDeltaMatrix)
{
// Copy each layer.
for (size_t i = 0; i < other.network.size(); ++i)
network.push_back(other.network[i]->Clone());
// Ensure that the aliases for layers during passes have the right size.
layerOutputs.resize(network.size(), MatType());
layerDeltas.resize(network.size(), MatType());
layerGradients.resize(network.size(), MatType());
// layerOutputs, layerDeltas, and layerGradients will be reset the next time
// Forward(), Backward(), or Gradient() is called.
}
template<typename MatType>
MultiLayer<MatType>::MultiLayer(MultiLayer&& other) :
Layer<MatType>(other),
network(std::move(other.network)),
inSize(std::move(other.inSize)),
totalInputSize(std::move(other.totalInputSize)),
totalOutputSize(std::move(other.totalOutputSize)),
layerOutputMatrix(std::move(other.layerOutputMatrix)),
layerDeltaMatrix(std::move(other.layerDeltaMatrix))
{
// Ensure that the aliases for layers during passes have the right size.
layerOutputs.resize(network.size(), MatType());
layerDeltas.resize(network.size(), MatType());
layerGradients.resize(network.size(), MatType());
// layerOutputs, layerDeltas, and layerGradients will be reset the next time
// Forward(), Backward(), or Gradient() is called.
other.layerOutputs.clear();
other.layerDeltas.clear();
other.layerGradients.clear();
}
template<typename MatType>
MultiLayer<MatType>& MultiLayer<MatType>::operator=(const MultiLayer& other)
{
if (this != &other)
{
Layer<MatType>::operator=(other);
network.clear();
layerOutputs.clear();
layerDeltas.clear();
layerGradients.clear();
inSize = other.inSize;
totalInputSize = other.totalInputSize;
totalOutputSize = other.totalOutputSize;
layerOutputMatrix = other.layerOutputMatrix;
layerDeltaMatrix = other.layerDeltaMatrix;
for (size_t i = 0; i < other.network.size(); ++i)
network.push_back(other.network[i]->Clone());
// Ensure that the aliases for layers during passes have the right size.
layerOutputs.resize(network.size(), MatType());
layerDeltas.resize(network.size(), MatType());
layerGradients.resize(network.size(), MatType());
}
return *this;
}
template<typename MatType>
MultiLayer<MatType>& MultiLayer<MatType>::operator=(MultiLayer&& other)
{
if (this != &other)
{
Layer<MatType>::operator=(other);
layerOutputs.clear();
layerDeltas.clear();
layerGradients.clear();
inSize = std::move(other.inSize);
totalInputSize = std::move(other.totalInputSize);
totalOutputSize = std::move(other.totalOutputSize);
network = std::move(other.network);
layerOutputs.resize(network.size(), MatType());
layerDeltas.resize(network.size(), MatType());
layerGradients.resize(network.size(), MatType());
other.layerOutputs.clear();
other.layerDeltas.clear();
other.layerGradients.clear();
}
return *this;
}
template<typename MatType>
void MultiLayer<MatType>::Forward(
const MatType& input, MatType& output)
{
Forward(input, output, 0, network.size() - 1);
}
template<typename MatType>
void MultiLayer<MatType>::Forward(
const MatType& input,
MatType& output,
const size_t start,
const size_t end)
{
// Make sure training/testing mode is set right in each layer.
for (size_t i = 0; i < network.size(); ++i)
network[i]->Training() = this->training;
// Note that we use `output` for the last layer; layerOutputs is only used for
// intermediate values between layers.
if ((end - start) > 0)
{
// Initialize memory for the forward pass (if needed).
InitializeForwardPassMemory(input.n_cols);
network[start]->Forward(input, layerOutputs[start]);
for (size_t i = start + 1; i < end; ++i)
network[i]->Forward(layerOutputs[i - 1], layerOutputs[i]);
network[end]->Forward(layerOutputs[end - 1], output);
}
else if ((end - start) == 0 && network.size() > 0)
{
network[start]->Forward(input, output);
}
else
{
// Empty network?
output = input;
}
}
template<typename MatType>
void MultiLayer<MatType>::Backward(
const MatType& input, const MatType& gy, MatType& g)
{
if (network.size() > 1)
{
// Initialize memory for the backward pass (if needed).
InitializeBackwardPassMemory(input.n_cols);
network.back()->Backward(input, gy, layerDeltas.back());
for (size_t i = network.size() - 2; i > 0; --i)
network[i]->Backward(layerOutputs[i], layerDeltas[i + 1], layerDeltas[i]);
network[0]->Backward(layerOutputs[0], layerDeltas[1], g);
}
else if (network.size() == 1)
{
network[0]->Backward(input, gy, g);
}
else
{
// Empty network?
g = input;
}
}
template<typename MatType>
void MultiLayer<MatType>::Gradient(
const MatType& input, const MatType& error, MatType& gradient)
{
// We assume gradient has the right size already.
// Pass gradients through each layer.
if (network.size() > 1)
{
// Initialize memory for the gradient pass (if needed).
InitializeGradientPassMemory(gradient);
network.front()->Gradient(input, layerDeltas[1], layerGradients.front());
for (size_t i = 1; i < network.size() - 1; ++i)
{
network[i]->Gradient(layerOutputs[i - 1], layerDeltas[i + 1],
layerGradients[i]);
}
network.back()->Gradient(layerOutputs[network.size() - 2], error,
layerGradients.back());
}
else if (network.size() == 1)
{
network[0]->Gradient(input, error, gradient);
}
else
{
// Nothing to do if the network is empty... there is no gradient.
}
}
template<typename MatType>
void MultiLayer<MatType>::SetWeights(typename MatType::elem_type* weightsPtr)
{
size_t start = 0;
const size_t totalWeightSize = WeightSize();
for (size_t i = 0; i < network.size(); ++i)
{
const size_t weightSize = network[i]->WeightSize();
// Sanity check: ensure we aren't passing memory past the end of the
// parameters.
Log::Assert(start + weightSize <= totalWeightSize,
"FNN::SetLayerMemory(): parameter size does not match total layer "
"weight size!");
network[i]->SetWeights(weightsPtr + start);
start += weightSize;
}
// Technically this check should be unnecessary, but there's nothing wrong
// with a little paranoia...
Log::Assert(start == totalWeightSize,
"FNN::SetLayerMemory(): total layer weight size does not match parameter "
"size!");
}
template<typename MatType>
size_t MultiLayer<MatType>::WeightSize() const
{
// Sum the weights in each layer.
size_t total = 0;
for (size_t i = 0; i < network.size(); ++i)
total += network[i]->WeightSize();
return total;
}
template<typename MatType>
void MultiLayer<MatType>::ComputeOutputDimensions()
{
inSize = 0;
totalInputSize = 0;
totalOutputSize = 0;
// Propagate the input dimensions forward to the output.
network.front()->InputDimensions() = this->inputDimensions;
inSize = this->inputDimensions[0];
for (size_t i = 1; i < this->inputDimensions.size(); ++i)
inSize *= this->inputDimensions[i];
totalInputSize += inSize;
for (size_t i = 1; i < network.size(); ++i)
{
network[i]->InputDimensions() = network[i - 1]->OutputDimensions();
size_t layerInputSize = network[i]->InputDimensions()[0];
for (size_t j = 1; j < network[i]->InputDimensions().size(); ++j)
layerInputSize *= network[i]->InputDimensions()[j];
totalInputSize += layerInputSize;
totalOutputSize += layerInputSize;
}
size_t lastLayerSize = network.back()->OutputDimensions()[0];
for (size_t i = 1; i < network.back()->OutputDimensions().size(); ++i)
lastLayerSize *= network.back()->OutputDimensions()[i];
totalOutputSize += lastLayerSize;
this->outputDimensions = network.back()->OutputDimensions();
}
template<typename MatType>
double MultiLayer<MatType>::Loss() const
{
double loss = 0.0;
for (size_t i = 0; i < network.size(); ++i)
loss += network[i]->Loss();
return loss;
}
template<typename MatType>
template<typename Archive>
void MultiLayer<MatType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(cereal::base_class<Layer<MatType>>(this));
ar(CEREAL_VECTOR_POINTER(network));
ar(CEREAL_NVP(inSize));
ar(CEREAL_NVP(totalInputSize));
ar(CEREAL_NVP(totalOutputSize));
if (Archive::is_loading::value)
{
layerOutputMatrix.clear();
layerDeltaMatrix.clear();
layerGradients.clear();
layerOutputs.resize(network.size(), MatType());
layerDeltas.resize(network.size(), MatType());
layerGradients.resize(network.size(), MatType());
}
}
template<typename MatType>
void MultiLayer<MatType>::InitializeForwardPassMemory(const size_t batchSize)
{
// We need to initialize memory to store the output of each layer's Forward()
// call. We'll do this all in one matrix, but, the size of this matrix
// depends on the batch size we are using for computation. We avoid resizing
// layerOutputMatrix down, unless we only need 10% or less of it.
if (batchSize * totalOutputSize > layerOutputMatrix.n_elem ||
batchSize * totalOutputSize <
std::floor(0.1 * layerOutputMatrix.n_elem))
{
// All outputs will be represented by one big block of memory.
layerOutputMatrix = MatType(1, batchSize * totalOutputSize);
}
// Now, create an alias to the right place for each layer. We assume that
// layerOutputs is already sized correctly (this should be done by Add()).
size_t start = 0;
for (size_t i = 0; i < layerOutputs.size(); ++i)
{
const size_t layerOutputSize = network[i]->OutputSize();
MakeAlias(layerOutputs[i], layerOutputMatrix.colptr(start),
layerOutputSize, batchSize);
start += batchSize * layerOutputSize;
}
}
template<typename MatType>
void MultiLayer<MatType>::InitializeBackwardPassMemory(
const size_t batchSize)
{
// We need to initialize memory to store the output of each layer's Backward()
// call. We do this similarly to InitializeForwardPassMemory(), but we must
// store a matrix to use as the delta for each layer.
if (batchSize * totalInputSize > layerDeltaMatrix.n_elem ||
batchSize * totalInputSize < std::floor(0.1 * layerDeltaMatrix.n_elem))
{
// All deltas will be represented by one big block of memory.
layerDeltaMatrix = MatType(1, batchSize * totalInputSize);
}
// Now, create an alias to the right place for each layer. We assume that
// layerDeltas is already sized correctly (this should be done by Add()).
size_t start = 0;
for (size_t i = 0; i < layerDeltas.size(); ++i)
{
size_t layerInputSize = 1;
if (i == 0)
{
for (size_t j = 0; j < this->inputDimensions.size(); ++j)
layerInputSize *= this->inputDimensions[j];
}
else
{
layerInputSize = network[i - 1]->OutputSize();
}
MakeAlias(layerDeltas[i], layerDeltaMatrix.colptr(start), layerInputSize,
batchSize);
start += batchSize * layerInputSize;
}
}
template<typename MatType>
void MultiLayer<MatType>::InitializeGradientPassMemory(MatType& gradient)
{
// We need to initialize memory to store the gradients of each layer. To do
// this, we need to know the weight size of each layer.
size_t gradientStart = 0;
for (size_t i = 0; i < network.size(); ++i)
{
const size_t weightSize = network[i]->WeightSize();
MakeAlias(layerGradients[i], gradient.memptr() + gradientStart,
weightSize, 1);
gradientStart += weightSize;
}
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/multiply_constant_impl.hpp
-96
View File
@@ -1,96 +0,0 @@
/**
* @file methods/ann/layer/multiply_constant_impl.hpp
* @author Marcus Edel
*
* Implementation of the MultiplyConstantLayer class, which multiplies the
* input by a (non-learnable) constant.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_MULTIPLY_CONSTANT_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_MULTIPLY_CONSTANT_IMPL_HPP
// In case it hasn't yet been included.
#include "multiply_constant.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
MultiplyConstant<InputDataType, OutputDataType>::MultiplyConstant(
const double scalar) : scalar(scalar)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
MultiplyConstant<InputDataType, OutputDataType>::MultiplyConstant(
const MultiplyConstant& layer) :
scalar(layer.scalar)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
MultiplyConstant<InputDataType, OutputDataType>::MultiplyConstant(
MultiplyConstant&& layer) :
scalar(std::move(layer.scalar))
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
MultiplyConstant<InputDataType, OutputDataType>&
MultiplyConstant<InputDataType, OutputDataType>::operator=(
const MultiplyConstant& layer)
{
if (this != &layer)
{
scalar = layer.scalar;
}
return *this;
}
template<typename InputDataType, typename OutputDataType>
MultiplyConstant<InputDataType, OutputDataType>&
MultiplyConstant<InputDataType, OutputDataType>::operator=(
MultiplyConstant&& layer)
{
if (this != &layer)
{
scalar = std::move(layer.scalar);
}
return *this;
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
void MultiplyConstant<InputDataType, OutputDataType>::Forward(
const InputType& input, OutputType& output)
{
output = input * scalar;
}
template<typename InputDataType, typename OutputDataType>
template<typename DataType>
void MultiplyConstant<InputDataType, OutputDataType>::Backward(
const DataType& /* input */, const DataType& gy, DataType& g)
{
g = gy * scalar;
}
template<typename InputDataType, typename OutputDataType>
template<typename Archive>
void MultiplyConstant<InputDataType, OutputDataType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(CEREAL_NVP(scalar));
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/multiply_merge_impl.hpp
-190
View File
@@ -1,190 +0,0 @@
/**
* @file methods/ann/layer/multiply_merge_impl.hpp
* @author Haritha Nair
*
* Definition of the MultiplyMerge module which multiplies the output of the
* given modules element-wise.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_MULTIPLY_MERGE_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_MULTIPLY_MERGE_IMPL_HPP
// In case it hasn't yet been included.
#include "multiply_merge.hpp"
#include "../visitor/forward_visitor.hpp"
#include "../visitor/backward_visitor.hpp"
#include "../visitor/gradient_visitor.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
MultiplyMerge<InputDataType, OutputDataType, CustomLayers...>::MultiplyMerge(
const bool model, const bool run) :
model(model), run(run), ownsLayer(!model)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
MultiplyMerge<InputDataType, OutputDataType, CustomLayers...>::MultiplyMerge(
const MultiplyMerge& layer) :
model(layer.model),
run(layer.run),
ownsLayer(layer.ownsLayer),
network(layer.network),
weights(layer.weights)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
MultiplyMerge<InputDataType, OutputDataType, CustomLayers...>::MultiplyMerge(
MultiplyMerge&& layer) :
model(std::move(layer.model)),
run(std::move(layer.run)),
ownsLayer(std::move(layer.ownsLayer)),
network(std::move(layer.network)),
weights(std::move(layer.weights))
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
MultiplyMerge<InputDataType, OutputDataType, CustomLayers...>&
MultiplyMerge<InputDataType, OutputDataType, CustomLayers...>::operator=(
const MultiplyMerge& layer)
{
if (this != &layer)
{
model = layer.model;
run = layer.run;
ownsLayer = layer.ownsLayer;
network = layer.network;
weights = layer.weights;
}
return *this;
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
MultiplyMerge<InputDataType, OutputDataType, CustomLayers...>&
MultiplyMerge<InputDataType, OutputDataType, CustomLayers...>::operator=(
MultiplyMerge&& layer)
{
if (this != &layer)
{
model = std::move(layer.model);
run = std::move(layer.run);
ownsLayer = std::move(layer.ownsLayer);
network = std::move(layer.network);
weights = std::move(layer.weights);
}
return *this;
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
MultiplyMerge<InputDataType, OutputDataType, CustomLayers...>::~MultiplyMerge()
{
if (ownsLayer)
{
std::for_each(network.begin(), network.end(),
boost::apply_visitor(deleteVisitor));
}
}
template <typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename InputType, typename OutputType>
void MultiplyMerge<InputDataType, OutputDataType, CustomLayers...>::Forward(
const InputType& input, OutputType& output)
{
if (run)
{
for (size_t i = 0; i < network.size(); ++i)
{
boost::apply_visitor(ForwardVisitor(input,
boost::apply_visitor(outputParameterVisitor, network[i])),
network[i]);
}
}
output = boost::apply_visitor(outputParameterVisitor, network.front());
for (size_t i = 1; i < network.size(); ++i)
{
output %= boost::apply_visitor(outputParameterVisitor, network[i]);
}
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void MultiplyMerge<InputDataType, OutputDataType, CustomLayers...>::Backward(
const arma::Mat<eT>& /* input */, const arma::Mat<eT>& gy, arma::Mat<eT>& g)
{
if (run)
{
for (size_t i = 0; i < network.size(); ++i)
{
boost::apply_visitor(BackwardVisitor(boost::apply_visitor(
outputParameterVisitor, network[i]), gy,
boost::apply_visitor(deltaVisitor, network[i])), network[i]);
}
g = boost::apply_visitor(deltaVisitor, network[0]);
for (size_t i = 1; i < network.size(); ++i)
{
g += boost::apply_visitor(deltaVisitor, network[i]);
}
}
else
g = gy;
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename eT>
void MultiplyMerge<InputDataType, OutputDataType, CustomLayers...>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& /* gradient */ )
{
if (run)
{
for (size_t i = 0; i < network.size(); ++i)
{
boost::apply_visitor(GradientVisitor(input, error), network[i]);
}
}
}
template<typename InputDataType, typename OutputDataType,
typename... CustomLayers>
template<typename Archive>
void MultiplyMerge<InputDataType, OutputDataType, CustomLayers...>::serialize(
Archive& ar, const uint32_t /* version */)
{
// Be sure to clear other layers before loading.
if (cereal::is_loading<Archive>())
network.clear();
ar(CEREAL_VECTOR_VARIANT_POINTER(network));
ar(CEREAL_NVP(model));
ar(CEREAL_NVP(run));
ar(CEREAL_NVP(ownsLayer));
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/noisylinear.hpp
+56 -105
View File
@@ -14,6 +14,8 @@
#include <mlpack/prereqs.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -21,55 +23,42 @@ namespace ann /** Artificial Neural Network. */ {
* Implementation of the NoisyLinear layer class. It represents a single
* layer of a neural network, with parametric noise added to its weights.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam MatType Matrix representation to accept as input and use for
* computation.
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class NoisyLinear
template<typename MatType = arma::mat>
class NoisyLinearType : public Layer<MatType>
{
public:
//! Create the NoisyLinear object.
NoisyLinear();
/**
* Create the NoisyLinear layer object using the specified number of units.
*
* @param inSize The number of input units.
* @param outSize The number of output units.
*/
NoisyLinear(const size_t inSize,
const size_t outSize);
NoisyLinearType(const size_t outSize = 0);
//! Copy constructor.
NoisyLinear(const NoisyLinear&);
//! Clone the NoisyLinearType object. This handles polymorphism correctly.
NoisyLinearType* Clone() const { return new NoisyLinearType(*this); }
//! Move constructor.
NoisyLinear(NoisyLinear&&);
// Virtual destructor.
virtual ~NoisyLinearType() { }
//! Operator= copy constructor.
NoisyLinear& operator=(const NoisyLinear& layer);
//! Copy the given NoisyLinear layer (but not weights).
NoisyLinearType(const NoisyLinearType& other);
//! Take ownership of the given NoisyLinear layer (but not weights).
NoisyLinearType(NoisyLinearType&& other);
//! Copy the given NoisyLinear layer (but not weights).
NoisyLinearType& operator=(const NoisyLinearType& other);
//! Take ownership of the given NoisyLinear layer (but not weights).
NoisyLinearType& operator=(NoisyLinearType&& other);
//! Operator= move constructor.
NoisyLinear& operator=(NoisyLinear&& layer);
//! Reset the layer parameter.
void SetWeights(typename MatType::elem_type* weightsPtr);
/*
* Reset the layer parameter.
*/
void Reset();
/*
* Reset the noise parameters(epsilons).
*/
//! Reset the noise parameters (epsilons).
void ResetNoise();
/*
* Reset the values of layer parameters (factorized gaussian noise).
*/
//! Reset the values of layer parameters (factorized gaussian noise).
void ResetParameters();
/**
@@ -79,8 +68,7 @@ class NoisyLinear
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const MatType& input, MatType& output);
/**
* Ordinary feed backward pass of a neural network, calculating the function
@@ -91,117 +79,80 @@ class NoisyLinear
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
void Backward(const MatType& /* input */,
const MatType& gy,
MatType& g);
/*
/**
* Calculate the gradient using the output delta and the input activation.
*
* @param input The input parameter used for calculating the gradient.
* @param error The calculated error.
* @param gradient The calculated gradient.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient);
void Gradient(const MatType& input,
const MatType& error,
MatType& gradient);
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
MatType const& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
//! Get the input parameter.
InputDataType const& InputParameter() const { return inputParameter; }
//! Modify the input parameter.
InputDataType& InputParameter() { return inputParameter; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the input size.
size_t InputSize() const { return inSize; }
//! Get the output size.
size_t OutputSize() const { return outSize; }
//! Get the gradient.
OutputDataType const& Gradient() const { return gradient; }
//! Modify the gradient.
OutputDataType& Gradient() { return gradient; }
MatType& Parameters() { return weights; }
//! Get the shape of the input.
size_t InputShape() const
{
return inSize;
}
//! Modify the bias weights of the layer.
arma::mat& Bias() { return bias; }
MatType& Bias() { return bias; }
//! Get size of weights.
//! Compute the number of parameters in the layer.
size_t WeightSize() const { return (outSize * inSize + outSize) * 2; }
/**
* Serialize the layer
*/
//! Compute the output dimensions of the layer given `InputDimensions()`.
void ComputeOutputDimensions();
//! Serialize the layer.
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Locally-stored number of input units.
size_t inSize;
//! Locally-stored number of output units.
size_t outSize;
//! Locally stored number of input units.
size_t inSize;
//! Locally-stored weight object.
OutputDataType weights;
MatType weights;
//! Locally-stored weight parameters.
OutputDataType weight;
MatType weight;
//! Locally-stored weight-mean parameters.
OutputDataType weightMu;
MatType weightMu;
//! Locally-stored weight-standard-deviation parameters.
OutputDataType weightSigma;
MatType weightSigma;
//! Locally-stored weight-epsilon parameters.
OutputDataType weightEpsilon;
MatType weightEpsilon;
//! Locally-stored bias parameters.
OutputDataType bias;
MatType bias;
//! Locally-stored bias-mean parameters.
OutputDataType biasMu;
MatType biasMu;
//! Locally-stored bias-standard-deviation parameters.
OutputDataType biasSigma;
MatType biasSigma;
//! Locally-stored bias-epsilon parameters.
OutputDataType biasEpsilon;
MatType biasEpsilon;
//! Locally-stored delta object.
OutputDataType delta;
}; // class NoisyLinearType
//! Locally-stored gradient object.
OutputDataType gradient;
// Convenience typedefs.
//! Locally-stored input parameter object.
InputDataType inputParameter;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
}; // class NoisyLinear
// Standard noisy linear layer.
typedef NoisyLinearType<arma::mat> NoisyLinear;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/noisylinear_impl.hpp
+85 -90
View File
@@ -18,107 +18,92 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
NoisyLinear<InputDataType, OutputDataType>::NoisyLinear() :
inSize(0),
outSize(0)
template<typename MatType>
NoisyLinearType<MatType>::NoisyLinearType(const size_t outSize) :
Layer<MatType>(),
outSize(outSize),
inSize(0)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
NoisyLinear<InputDataType, OutputDataType>::NoisyLinear(
const NoisyLinear& layer) :
inSize(layer.inSize),
outSize(layer.outSize),
weights(layer.weights)
template<typename MatType>
NoisyLinearType<MatType>::NoisyLinearType(const NoisyLinearType& other) :
Layer<MatType>(other),
outSize(other.outSize),
inSize(other.inSize)
{
Reset();
// Nothing to do.
}
template<typename InputDataType, typename OutputDataType>
NoisyLinear<InputDataType, OutputDataType>::NoisyLinear(
const size_t inSize,
const size_t outSize) :
inSize(inSize),
outSize(outSize)
template<typename MatType>
NoisyLinearType<MatType>::NoisyLinearType(NoisyLinearType&& other) :
Layer<MatType>(std::move(other)),
outSize(std::move(other.outSize)),
inSize(std::move(other.inSize))
{
weights.set_size(WeightSize(), 1);
weightEpsilon.set_size(outSize, inSize);
biasEpsilon.set_size(outSize, 1);
// Nothing to do.
}
template<typename InputDataType, typename OutputDataType>
NoisyLinear<InputDataType, OutputDataType>::NoisyLinear(
NoisyLinear&& layer) :
inSize(std::move(layer.inSize)),
outSize(std::move(layer.outSize)),
weights(std::move(layer.weights))
template<typename MatType>
NoisyLinearType<MatType>&
NoisyLinearType<MatType>::operator=(const NoisyLinearType& other)
{
layer.inSize = 0;
layer.outSize = 0;
layer.weights = nullptr;
Reset();
}
template<typename InputDataType, typename OutputDataType>
NoisyLinear<InputDataType, OutputDataType>&
NoisyLinear<InputDataType, OutputDataType>::operator=(const NoisyLinear& layer)
{
if (this != &layer)
if (&other != this)
{
inSize = layer.inSize;
outSize = layer.outSize;
weights = layer.weights;
Reset();
Layer<MatType>::operator=(other);
outSize = other.outSize;
inSize = other.inSize;
}
return *this;
}
template<typename InputDataType, typename OutputDataType>
NoisyLinear<InputDataType, OutputDataType>&
NoisyLinear<InputDataType, OutputDataType>::operator=(NoisyLinear&& layer)
template<typename MatType>
NoisyLinearType<MatType>&
NoisyLinearType<MatType>::operator=(NoisyLinearType&& other)
{
if (this != &layer)
if (&other != this)
{
inSize = std::move(layer.inSize);
layer.inSize = 0;
outSize = std::move(layer.outSize);
layer.outSize = 0;
weights = std::move(layer.weights);
layer.weights = nullptr;
Reset();
Layer<MatType>::operator=(std::move(other));
outSize = std::move(other.outSize);
inSize = std::move(other.inSize);
}
return *this;
}
template<typename InputDataType, typename OutputDataType>
void NoisyLinear<InputDataType, OutputDataType>::Reset()
template<typename MatType>
void NoisyLinearType<MatType>::SetWeights(
typename MatType::elem_type* weightsPtr)
{
weightMu = arma::mat(weights.memptr(),
outSize, inSize, false, false);
biasMu = arma::mat(weights.memptr() + weightMu.n_elem,
outSize, 1, false, false);
weightSigma = arma::mat(weights.memptr() + weightMu.n_elem + biasMu.n_elem,
outSize, inSize, false, false);
biasSigma = arma::mat(weights.memptr() + weightMu.n_elem * 2 + biasMu.n_elem,
outSize, 1, false, false);
MakeAlias(weights, weightsPtr, 1, (outSize * inSize + outSize) * 2);
MakeAlias(weightMu, weightsPtr, outSize, inSize);
MakeAlias(biasMu, weightsPtr + weightMu.n_elem, outSize, 1);
MakeAlias(weightSigma, weightsPtr + weightMu.n_elem + biasMu.n_elem, outSize,
inSize);
MakeAlias(biasSigma, weightsPtr + weightMu.n_elem * 2 + biasMu.n_elem,
outSize, 1);
this->ResetNoise();
}
template<typename InputDataType, typename OutputDataType>
void NoisyLinear<InputDataType, OutputDataType>::ResetNoise()
template<typename MatType>
void NoisyLinearType<MatType>::ResetNoise()
{
arma::mat epsilonIn = arma::randn<arma::mat>(inSize, 1);
MatType epsilonIn = arma::randn<MatType>(inSize, 1);
epsilonIn = arma::sign(epsilonIn) % arma::sqrt(arma::abs(epsilonIn));
arma::mat epsilonOut = arma::randn<arma::mat>(outSize, 1);
MatType epsilonOut = arma::randn<MatType>(outSize, 1);
epsilonOut = arma::sign(epsilonOut) % arma::sqrt(arma::abs(epsilonOut));
weightEpsilon = epsilonOut * epsilonIn.t();
biasEpsilon = epsilonOut;
}
template<typename InputDataType, typename OutputDataType>
void NoisyLinear<InputDataType, OutputDataType>::ResetParameters()
template<typename MatType>
void NoisyLinearType<MatType>::ResetParameters()
{
const double muRange = 1 / std::sqrt(inSize);
weightMu.randu();
@@ -129,10 +114,8 @@ void NoisyLinear<InputDataType, OutputDataType>::ResetParameters()
biasSigma.fill(0.5 / std::sqrt(outSize));
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void NoisyLinear<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
template<typename MatType>
void NoisyLinearType<MatType>::Forward(const MatType& input, MatType& output)
{
weight = weightMu + weightSigma % weightEpsilon;
bias = biasMu + biasSigma % biasEpsilon;
@@ -140,23 +123,19 @@ void NoisyLinear<InputDataType, OutputDataType>::Forward(
output.each_col() += bias;
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void NoisyLinear<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& /* input */, const arma::Mat<eT>& gy, arma::Mat<eT>& g)
template<typename MatType>
void NoisyLinearType<MatType>::Backward(
const MatType& /* input */, const MatType& gy, MatType& g)
{
g = weight.t() * gy;
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void NoisyLinear<InputDataType, OutputDataType>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient)
template<typename MatType>
void NoisyLinearType<MatType>::Gradient(
const MatType& input, const MatType& error, MatType& gradient)
{
// Locally stored to prevent multiplication twice.
arma::mat weightGrad = error * input.t();
MatType weightGrad = error * input.t();
// Gradients for mu values.
gradient.rows(0, weight.n_elem - 1) = arma::vectorise(weightGrad);
@@ -170,18 +149,34 @@ void NoisyLinear<InputDataType, OutputDataType>::Gradient(
= arma::sum(error, 1) % biasEpsilon;
}
template<typename InputDataType, typename OutputDataType>
template<typename MatType>
void NoisyLinearType<MatType>::ComputeOutputDimensions()
{
inSize = this->inputDimensions[0];
for (size_t i = 1; i < this->inputDimensions.size(); ++i)
inSize *= this->inputDimensions[i];
this->outputDimensions = std::vector<size_t>(this->inputDimensions.size(),
1);
// The NoisyLinear layer flattens its output.
this->outputDimensions[0] = outSize;
}
template<typename MatType>
template<typename Archive>
void NoisyLinear<InputDataType, OutputDataType>::serialize(
void NoisyLinearType<MatType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(CEREAL_NVP(inSize));
ar(CEREAL_NVP(outSize));
ar(cereal::base_class<Layer<MatType>>(this));
ar(CEREAL_NVP(outSize));
ar(CEREAL_NVP(inSize));
// This is inefficient, but we have to allocate this memory so that
// WeightSetVisitor gets the right size.
if (cereal::is_loading<Archive>())
weights.set_size((outSize * inSize + outSize) * 2, 1);
{
ResetNoise();
}
}
} // namespace ann
src/mlpack/methods/ann/layer/not_adapted/README.md
+9
View File
@@ -0,0 +1,9 @@
Layers in this directory were written with the old boost::visitor interface. In
[#2777](https://github.com/mlpack/mlpack/pull/2777), we adapted each layer to
use inheritance instead. However, time did not permit the adaptation of all
layers, and so remaining layers that have not yet been adapted are in this
directory.
The intention is that we will work our way through layers in this directory,
updating them to the new interface and re-enabling tests for them in separate,
follow-up PRs. If you'd like to help out, you are more than welcome to!
src/mlpack/methods/ann/layer/adaptive_max_pooling.hpp → src/mlpack/methods/ann/layer/not_adapted/adaptive_max_pooling.hpp
+38 -52
View File
@@ -13,7 +13,9 @@
#define MLPACK_METHODS_ANN_LAYER_ADAPTIVE_MAX_POOLING_HPP
#include <mlpack/prereqs.hpp>
#include "layer_types.hpp"
#include "layer.hpp"
#include "max_pooling.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -21,20 +23,17 @@ namespace ann /** Artificial Neural Network. */ {
/**
* Implementation of the AdaptiveMaxPooling layer.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam InputType The type of the layer's inputs. The layer automatically
* cast inputs to this type (Default: arma::mat).
* @tparam OutputType The type of the layer's Outputs. The layer automatically
* cast inputs to this type (Default: arma::mat).
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class AdaptiveMaxPooling
template <typename InputType = arma::mat, typename OutputType = arma::mat>
class AdaptiveMaxPoolingType : public Layer<InputType, OutputType>
{
public:
//! Create the AdaptiveMaxPooling object.
AdaptiveMaxPooling();
AdaptiveMaxPoolingType();
/**
* Create the AdaptiveMaxPooling object.
@@ -42,15 +41,16 @@ class AdaptiveMaxPooling
* @param outputWidth Width of the output.
* @param outputHeight Height of the output.
*/
AdaptiveMaxPooling(const size_t outputWidth,
const size_t outputHeight);
AdaptiveMaxPoolingType(const size_t outputWidth,
const size_t outputHeight);
/**
* Create the AdaptiveMaxPooling object.
*
* @param outputShape A two-value tuple indicating width and height of the output.
* @param outputShape A two-value tuple indicating width and height of the
* output.
*/
AdaptiveMaxPooling(const std::tuple<size_t, size_t>& outputShape);
AdaptiveMaxPoolingType(const std::tuple<size_t, size_t>& outputShape);
/**
* Ordinary feed forward pass of a neural network, evaluating the function
@@ -59,8 +59,7 @@ class AdaptiveMaxPooling
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const InputType& input, OutputType& output);
/**
* Ordinary feed backward pass of a neural network, using 3rd-order tensors as
@@ -71,48 +70,30 @@ class AdaptiveMaxPooling
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& input,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
//! Get the output parameter.
const OutputDataType& OutputParameter() const
{ return poolingLayer.OutputParameter(); }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return poolingLayer.OutputParameter(); }
//! Get the delta.
const OutputDataType& Delta() const { return poolingLayer.Delta(); }
//! Modify the delta.
OutputDataType& Delta() { return poolingLayer.Delta(); }
//! Get the input width.
size_t InputWidth() const { return poolingLayer.InputWidth(); }
//! Modify the input width.
size_t& InputWidth() { return poolingLayer.InputWidth(); }
//! Get the input height.
size_t InputHeight() const { return poolingLayer.InputHeight(); }
//! Modify the input height.
size_t& InputHeight() { return poolingLayer.InputHeight(); }
void Backward(const InputType& input,
const OutputType& gy,
OutputType& g);
//! Get the output width.
size_t OutputWidth() const { return outputWidth; }
size_t const& OutputWidth() const { return outputWidth; }
//! Modify the output width.
size_t& OutputWidth() { return outputWidth; }
//! Get the output height.
size_t OutputHeight() const { return outputHeight; }
size_t const& OutputHeight() const { return outputHeight; }
//! Modify the output height.
size_t& OutputHeight() { return outputHeight; }
//! Get the input size.
size_t InputSize() const { return poolingLayer.InputSize(); }
//! Get the number of trainable weights.
size_t WeightSize() const { return 0; }
//! Get the output size.
size_t OutputSize() const { return poolingLayer.OutputSize(); }
const std::vector<size_t>& OutputDimensions() const
{
std::vector<size_t> result(this->inputDimensions.size(), 1);
result[0] = outputWidth;
result[1] = outputHeight;
return result;
}
//! Get the size of the weights.
size_t WeightSize() const { return 0; }
@@ -127,7 +108,7 @@ class AdaptiveMaxPooling
/**
* Initialize Kernel Size and Stride for Adaptive Pooling.
*/
void IntializeAdaptivePadding()
void InitializeAdaptivePadding()
{
poolingLayer.StrideWidth() = std::floor(poolingLayer.InputWidth() /
outputWidth);
@@ -150,7 +131,7 @@ class AdaptiveMaxPooling
}
//! Locally stored MaxPooling Object.
MaxPooling<InputDataType, OutputDataType> poolingLayer;
MaxPoolingType<InputType, OutputType> poolingLayer;
//! Locally-stored output width.
size_t outputWidth;
@@ -160,7 +141,12 @@ class AdaptiveMaxPooling
//! Locally-stored reset parameter used to initialize the layer once.
bool reset;
}; // class AdaptiveMaxPooling
}; // class AdaptiveMaxPoolingType
// Convenience typedefs.
// Standard Adaptive max pooling layer.
typedef AdaptiveMaxPoolingType<arma::mat, arma::mat> AdaptiveMaxPooling;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/adaptive_max_pooling_impl.hpp → src/mlpack/methods/ann/layer/not_adapted/adaptive_max_pooling_impl.hpp
+21 -21
View File
@@ -18,61 +18,61 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
AdaptiveMaxPooling<InputDataType, OutputDataType>::AdaptiveMaxPooling()
template<typename InputType, typename OutputType>
AdaptiveMaxPoolingType<InputType, OutputType>::AdaptiveMaxPoolingType()
{
// Nothing to do here.
}
template <typename InputDataType, typename OutputDataType>
AdaptiveMaxPooling<InputDataType, OutputDataType>::AdaptiveMaxPooling(
template <typename InputType, typename OutputType>
AdaptiveMaxPoolingType<InputType, OutputType>::AdaptiveMaxPoolingType(
const size_t outputWidth,
const size_t outputHeight) :
AdaptiveMaxPooling(std::tuple<size_t, size_t>(outputWidth, outputHeight))
AdaptiveMaxPoolingType(std::tuple<size_t, size_t>(outputWidth, outputHeight))
{
// Nothing to do here.
}
template <typename InputDataType, typename OutputDataType>
AdaptiveMaxPooling<InputDataType, OutputDataType>::AdaptiveMaxPooling(
template <typename InputType, typename OutputType>
AdaptiveMaxPoolingType<InputType, OutputType>::AdaptiveMaxPoolingType(
const std::tuple<size_t, size_t>& outputShape):
outputWidth(std::get<0>(outputShape)),
outputHeight(std::get<1>(outputShape)),
reset(false)
{
poolingLayer = ann::MaxPooling<>(0, 0);
poolingLayer = ann::MaxPoolingType<InputType, OutputType>(0, 0);
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void AdaptiveMaxPooling<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
template<typename InputType, typename OutputType>
void AdaptiveMaxPoolingType<InputType, OutputType>::Forward(
const InputType& input, OutputType& output)
{
if (!reset)
{
IntializeAdaptivePadding();
InitializeAdaptivePadding();
reset = true;
}
poolingLayer.Forward(input, output);
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void AdaptiveMaxPooling<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& input,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g)
template<typename InputType, typename OutputType>
void AdaptiveMaxPoolingType<InputType, OutputType>::Backward(
const InputType& input,
const OutputType& gy,
OutputType& g)
{
poolingLayer.Backward(input, gy, g);
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
template<typename Archive>
void AdaptiveMaxPooling<InputDataType, OutputDataType>::serialize(
void AdaptiveMaxPoolingType<InputType, OutputType>::serialize(
Archive& ar,
const uint32_t /* version */)
{
ar(cereal::base_class<Layer<InputType, OutputType>>(this));
ar(CEREAL_NVP(outputWidth));
ar(CEREAL_NVP(outputHeight));
ar(CEREAL_NVP(reset));
src/mlpack/methods/ann/layer/adaptive_mean_pooling.hpp → src/mlpack/methods/ann/layer/not_adapted/adaptive_mean_pooling.hpp
+39 -52
View File
@@ -1,3 +1,4 @@
// Maybe
/**
* @file methods/ann/layer/adaptive_mean_pooling.hpp
* @author Kartik Dutt
@@ -14,7 +15,9 @@
#define MLPACK_METHODS_ANN_LAYER_ADAPTIVE_MEAN_POOLING_HPP
#include <mlpack/prereqs.hpp>
#include "layer_types.hpp"
#include "layer.hpp"
#include "mean_pooling.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -22,20 +25,17 @@ namespace ann /** Artificial Neural Network. */ {
/**
* Implementation of the AdaptiveMeanPooling.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam InputType The type of the layer's inputs. The layer automatically
* cast inputs to this type (Default: arma::mat).
* @tparam OutputType The type of the layer's Outputs. The layer automatically
* cast inputs to this type (Default: arma::mat).
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class AdaptiveMeanPooling
template <typename InputType = arma::mat, typename OutputType = arma::mat>
class AdaptiveMeanPoolingType : public Layer<InputType, OutputType>
{
public:
//! Create the AdaptiveMeanPooling object.
AdaptiveMeanPooling();
AdaptiveMeanPoolingType();
/**
* Create the AdaptiveMeanPooling object.
@@ -43,15 +43,16 @@ class AdaptiveMeanPooling
* @param outputWidth Width of the output.
* @param outputHeight Height of the output.
*/
AdaptiveMeanPooling(const size_t outputWidth,
const size_t outputHeight);
AdaptiveMeanPoolingType(const size_t outputWidth,
const size_t outputHeight);
/**
* Create the AdaptiveMeanPooling object.
*
* @param outputShape A two-value tuple indicating width and height of the output.
* @param outputShape A two-value tuple indicating width and height of the
* output.
*/
AdaptiveMeanPooling(const std::tuple<size_t, size_t>& outputShape);
AdaptiveMeanPoolingType(const std::tuple<size_t, size_t>& outputShape);
/**
* Ordinary feed forward pass of a neural network, evaluating the function
@@ -60,8 +61,7 @@ class AdaptiveMeanPooling
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const InputType& input, OutputType& output);
/**
* Ordinary feed backward pass of a neural network, using 3rd-order tensors as
@@ -72,48 +72,30 @@ class AdaptiveMeanPooling
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& input,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
//! Get the output parameter.
const OutputDataType& OutputParameter() const
{ return poolingLayer.OutputParameter(); }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return poolingLayer.OutputParameter(); }
//! Get the delta.
const OutputDataType& Delta() const { return poolingLayer.Delta(); }
//! Modify the delta.
OutputDataType& Delta() { return poolingLayer.Delta(); }
//! Get the input width.
size_t InputWidth() const { return poolingLayer.InputWidth(); }
//! Modify the input width.
size_t& InputWidth() { return poolingLayer.InputWidth(); }
//! Get the input height.
size_t InputHeight() const { return poolingLayer.InputHeight(); }
//! Modify the input height.
size_t& InputHeight() { return poolingLayer.InputHeight(); }
void Backward(const InputType& input,
const OutputType& gy,
OutputType& g);
//! Get the output width.
size_t OutputWidth() const { return outputWidth; }
size_t const& OutputWidth() const { return outputWidth; }
//! Modify the output width.
size_t& OutputWidth() { return outputWidth; }
//! Get the output height.
size_t OutputHeight() const { return outputHeight; }
size_t const& OutputHeight() const { return outputHeight; }
//! Modify the output height.
size_t& OutputHeight() { return outputHeight; }
//! Get the input size.
size_t InputSize() const { return poolingLayer.InputSize(); }
//! Get the number of trainable weights.
size_t WeightSize() const { return 0; }
//! Get the output size.
size_t OutputSize() const { return poolingLayer.OutputSize(); }
const std::vector<size_t>& OutputDimensions() const
{
std::vector<size_t> result(this->inputDimensions);
result[0] = outputWidth;
result[1] = outputHeight;
return result;
}
//! Get the size of the weights.
size_t WeightSize() const { return 0; }
@@ -128,7 +110,7 @@ class AdaptiveMeanPooling
/**
* Initialize Kernel Size and Stride for Adaptive Pooling.
*/
void IntializeAdaptivePadding()
void InitializeAdaptivePadding()
{
poolingLayer.StrideWidth() = std::floor(poolingLayer.InputWidth() /
outputWidth);
@@ -151,7 +133,7 @@ class AdaptiveMeanPooling
}
//! Locally stored MeanPooling Object.
MeanPooling<InputDataType, OutputDataType> poolingLayer;
MeanPoolingType<InputType, OutputType> poolingLayer;
//! Locally-stored output width.
size_t outputWidth;
@@ -161,7 +143,12 @@ class AdaptiveMeanPooling
//! Locally-stored reset parameter used to initialize the layer once.
bool reset;
}; // class AdaptiveMeanPooling
}; // class AdaptiveMeanPoolingType
// Convenience typedefs.
// Standard Adaptive mean pooling layer.
typedef AdaptiveMeanPoolingType<arma::mat, arma::mat> AdaptiveMeanPooling;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/adaptive_mean_pooling_impl.hpp → src/mlpack/methods/ann/layer/not_adapted/adaptive_mean_pooling_impl.hpp
+21 -21
View File
@@ -18,61 +18,61 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
AdaptiveMeanPooling<InputDataType, OutputDataType>::AdaptiveMeanPooling()
template<typename InputType, typename OutputType>
AdaptiveMeanPoolingType<InputType, OutputType>::AdaptiveMeanPoolingType()
{
// Nothing to do here.
}
template <typename InputDataType, typename OutputDataType>
AdaptiveMeanPooling<InputDataType, OutputDataType>::AdaptiveMeanPooling(
template <typename InputType, typename OutputType>
AdaptiveMeanPoolingType<InputType, OutputType>::AdaptiveMeanPoolingType(
const size_t outputWidth,
const size_t outputHeight) :
AdaptiveMeanPooling(std::tuple<size_t, size_t>(outputWidth, outputHeight))
AdaptiveMeanPoolingType(std::tuple<size_t, size_t>(outputWidth, outputHeight))
{
// Nothing to do here.
}
template <typename InputDataType, typename OutputDataType>
AdaptiveMeanPooling<InputDataType, OutputDataType>::AdaptiveMeanPooling(
template <typename InputType, typename OutputType>
AdaptiveMeanPoolingType<InputType, OutputType>::AdaptiveMeanPoolingType(
const std::tuple<size_t, size_t>& outputShape):
outputWidth(std::get<0>(outputShape)),
outputHeight(std::get<1>(outputShape)),
reset(false)
{
poolingLayer = ann::MeanPooling<>(0, 0);
poolingLayer = ann::MeanPoolingType<InputType, OutputType>(0, 0);
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void AdaptiveMeanPooling<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
template<typename InputType, typename OutputType>
void AdaptiveMeanPoolingType<InputType, OutputType>::Forward(
const InputType& input, OutputType& output)
{
if (!reset)
{
IntializeAdaptivePadding();
InitializeAdaptivePadding();
reset = true;
}
poolingLayer.Forward(input, output);
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void AdaptiveMeanPooling<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& input,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g)
template<typename InputType, typename OutputType>
void AdaptiveMeanPoolingType<InputType, OutputType>::Backward(
const InputType& input,
const OutputType& gy,
OutputType& g)
{
poolingLayer.Backward(input, gy, g);
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
template<typename Archive>
void AdaptiveMeanPooling<InputDataType, OutputDataType>::serialize(
void AdaptiveMeanPoolingType<InputType, OutputType>::serialize(
Archive& ar,
const uint32_t /* version */)
{
ar(cereal::base_class<Layer<InputType, OutputType>>(this));
ar(CEREAL_NVP(outputWidth));
ar(CEREAL_NVP(outputHeight));
ar(CEREAL_NVP(reset));
src/mlpack/methods/ann/layer/add_merge.hpp → src/mlpack/methods/ann/layer/not_adapted/add_merge.hpp
+29 -109
View File
@@ -1,3 +1,4 @@
// Temporarily drop.
/**
* @file methods/ann/layer/add_merge.hpp
* @author Marcus Edel
@@ -15,10 +16,6 @@
#include <mlpack/prereqs.hpp>
#include "../visitor/delete_visitor.hpp"
#include "../visitor/delta_visitor.hpp"
#include "../visitor/output_parameter_visitor.hpp"
#include "layer_types.hpp"
namespace mlpack {
@@ -28,18 +25,17 @@ namespace ann /** Artificial Neural Network. */ {
* Implementation of the AddMerge module class. The AddMerge class accumulates
* the output of various modules.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* @tparam InputType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* @tparam OutputType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam CustomLayers Additional custom layers that can be added.
*/
template<
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat,
typename... CustomLayers
typename InputType = arma::mat,
typename OutputType = arma::mat
>
class AddMerge
class AddMerge : public MultiLayer<InputType, OutputType>
{
public:
/**
@@ -48,7 +44,7 @@ class AddMerge
* @param model Expose all the network modules.
* @param run Call the Forward/Backward method before the output is merged.
*/
AddMerge(const bool model = false, const bool run = true);
AddMerge(const bool run = true);
/**
* Create the AddMerge object using the specified parameters.
@@ -57,7 +53,7 @@ class AddMerge
* @param run Call the Forward/Backward method before the output is merged.
* @param ownsLayers Delete the layers when this is deallocated.
*/
AddMerge(const bool model, const bool run, const bool ownsLayers);
AddMerge(const bool run, const bool ownsLayers);
//! Destructor to release allocated memory.
~AddMerge();
@@ -69,8 +65,7 @@ class AddMerge
* @param * (input) Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename InputType, typename OutputType>
void Forward(const InputType& /* input */, OutputType& output);
void Forward(const InputType& input, OutputType& output);
/**
* Ordinary feed backward pass of a neural network, calculating the function
@@ -81,10 +76,9 @@ class AddMerge
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
void Backward(const InputType& /* input */,
const OutputType& gy,
OutputType& g);
/**
* This is the overload of Backward() that runs only a specific layer with
@@ -95,10 +89,9 @@ class AddMerge
* @param g The calculated gradient.
* @param index The index of the layer to run.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g,
void Backward(const InputType& /* input */,
const OutputType& gy,
OutputType& g,
const size_t index);
/*
@@ -108,10 +101,9 @@ class AddMerge
* @param error The calculated error.
* @param gradient The calculated gradient.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient);
void Gradient(const InputType& input,
const OutputType& error,
OutputType& gradient);
/*
* This is the overload of Gradient() that runs a specific layer with the
@@ -122,63 +114,24 @@ class AddMerge
* @param gradient The calculated gradient.
* @param The index of the layer to run.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient,
void Gradient(const InputType& input,
const OutputType& error,
OutputType& gradient,
const size_t index);
/*
* Add a new module to the model.
*
* @param args The layer parameter.
*/
template <class LayerType, class... Args>
void Add(Args... args) { network.push_back(new LayerType(args...)); }
/*
* Add a new module to the model.
*
* @param layer The Layer to be added to the model.
*/
void Add(LayerTypes<CustomLayers...> layer) { network.push_back(layer); }
//! Get the input parameter.
InputDataType const& InputParameter() const { return inputParameter; }
//! Modify the input parameter.
InputDataType& InputParameter() { return inputParameter; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Return the model modules.
std::vector<LayerTypes<CustomLayers...> >& Model()
{
if (model)
{
return network;
}
return empty;
}
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
//! Get the value of run parameter.
bool Run() const { return run; }
//! Modify the value of run parameter.
bool& Run() { return run; }
const std::vector<size_t>& OutputDimensions() const
{
// Propagate input size to child layers.
for (size_t i = 0; i < this->network.size(); ++i)
this->network[i]->InputDimensions() = this->inputDimensions;
return this->network.back()->OutputDimensions();
}
/**
* Serialize the layer.
*/
@@ -186,9 +139,6 @@ class AddMerge
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Parameter which indicates if the modules should be exposed.
bool model;
//! Parameter which indicates if the Forward/Backward method should be called
//! before merging the output.
bool run;
@@ -196,36 +146,6 @@ class AddMerge
//! We need this to know whether we should delete the internally-held layers
//! in the destructor.
bool ownsLayers;
//! Locally-stored network modules.
std::vector<LayerTypes<CustomLayers...> > network;
//! Locally-stored empty list of modules.
std::vector<LayerTypes<CustomLayers...> > empty;
//! Locally-stored delete visitor module object.
DeleteVisitor deleteVisitor;
//! Locally-stored output parameter visitor module object.
OutputParameterVisitor outputParameterVisitor;
//! Locally-stored delta visitor module object.
DeltaVisitor deltaVisitor;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored gradient object.
OutputDataType gradient;
//! Locally-stored input parameter object.
InputDataType inputParameter;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
//! Locally-stored weight object.
OutputDataType weights;
}; // class AddMerge
} // namespace ann
src/mlpack/methods/ann/layer/not_adapted/add_merge_impl.hpp
+145
View File
@@ -0,0 +1,145 @@
/**
* @file methods/ann/layer/add_merge_impl.hpp
* @author Marcus Edel
*
* Definition of the AddMerge module which accumulates the output of the given
* modules.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_ADD_MERGE_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_ADD_MERGE_IMPL_HPP
// In case it hasn't yet been included.
#include "add_merge.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputType, typename OutputType>
AddMerge<InputType, OutputType>::AddMerge(
const bool run) :
run(run), ownsLayers(true)
{
// Nothing to do here.
}
template<typename InputType, typename OutputType>
AddMerge<InputType, OutputType>::AddMerge(
const bool run, const bool ownsLayers) :
run(run), ownsLayers(ownsLayers)
{
// Nothing to do here.
}
template<typename InputType, typename OutputType>
AddMerge<InputType, OutputType>::~AddMerge()
{
}
template<typename InputType, typename OutputType>
void AddMerge<InputType, OutputType>::Forward(
const InputType& input, OutputType& output)
{
this->InitializeForwardPassMemory();
if (run)
{
for (size_t i = 0; i < this->network.size(); ++i)
{
this->network[i]->Forward(input, this->layerOutputs[i]);
}
}
output = this->layerOutputs.front();
for (size_t i = 1; i < this->network.size(); ++i)
{
output += this->layerOutputs[i];
}
}
template<typename InputType, typename OutputType>
void AddMerge<InputType, OutputType>::Backward(
const InputType& /* input */,
const OutputType& gy,
OutputType& g)
{
this->InitializeBackwardPassMemory();
if (run)
{
for (size_t i = 0; i < this->network.size(); ++i)
{
this->network[i]->Backward(this->layerOutputs[i], gy,
this->layerDeltas[i]);
}
g = this->layerDeltas[0];
for (size_t i = 1; i < this->network.size(); ++i)
{
g += this->layerDeltas[i];
}
}
else
{
g = gy;
}
}
template<typename InputType, typename OutputType>
void AddMerge<InputType, OutputType>::Backward(
const InputType& /* input */,
const OutputType& gy,
OutputType& g,
const size_t index)
{
this->network[index]->Backward(this->layerOutputs[index], gy, g);
}
template<typename InputType, typename OutputType>
void AddMerge<InputType, OutputType>::Gradient(
const InputType& input,
const OutputType& error,
OutputType& gradient)
{
if (run)
{
size_t start = 0;
for (size_t i = 0; i < this->network.size(); ++i)
{
this->network[i]->Gradient(input, error, OutputType(gradient.colptr(start),
1, this->network[i]->WeightSize(), false, true));
start += this->network[i]->WeightSize();
}
}
}
template<typename InputType, typename OutputType>
void AddMerge<InputType, OutputType>::Gradient(
const InputType& input,
const OutputType& error,
OutputType& gradient,
const size_t index)
{
this->network[index]->Gradient(input, error, gradient);
}
template<typename InputType, typename OutputType>
template<typename Archive>
void AddMerge<InputType, OutputType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(cereal::base_class<MultiLayer<InputType, OutputType>>(this));
ar(CEREAL_NVP(run));
ar(CEREAL_NVP(ownsLayers));
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/atrous_convolution.hpp → src/mlpack/methods/ann/layer/not_adapted/atrous_convolution.hpp
+48 -69
View File
@@ -1,3 +1,4 @@
// Temporarily drop.
/**
* @file methods/ann/layer/atrous_convolution.hpp
* @author Aarush Gupta
@@ -46,10 +47,10 @@ template <
typename ForwardConvolutionRule = NaiveConvolution<ValidConvolution>,
typename BackwardConvolutionRule = NaiveConvolution<FullConvolution>,
typename GradientConvolutionRule = NaiveConvolution<ValidConvolution>,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
typename InputType = arma::mat,
typename OutputType = arma::mat
>
class AtrousConvolution
class AtrousConvolution : public Layer<InputType, OutputType>
{
public:
//! Create the AtrousConvolution object.
@@ -139,8 +140,7 @@ class AtrousConvolution
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const InputType& input, OutputType& output);
/**
* Ordinary feed backward pass of a neural network, calculating the function
@@ -151,10 +151,9 @@ class AtrousConvolution
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
void Backward(const InputType& /* input */,
const OutputType& gy,
OutputType& g);
/*
* Calculate the gradient using the output delta and the input activation.
@@ -163,101 +162,90 @@ class AtrousConvolution
* @param error The calculated error.
* @param gradient The calculated gradient.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient);
void Gradient(const InputType& /* input */,
const OutputType& error,
OutputType& gradient);
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
OutputType const& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
OutputType& Parameters() { return weights; }
//! Get the weight of the layer.
arma::cube const& Weight() const { return weight; }
const arma::Cube<typename OutputType::elem_type>& Weight() const
{
return weight;
}
//! Modify the weight of the layer.
arma::cube& Weight() { return weight; }
arma::Cube<typename OutputType::elem_type>& Weight() { return weight; }
const std::vector<size_t>& OutputDimensions() const
{
std::vector<size_t> result(inputDimensions.size(), 0);
result[0] = outputWidth;
result[1] = outputHeight;
return result;
}
//! Get the bias of the layer.
arma::mat const& Bias() const { return bias; }
const OutputType& Bias() const { return bias; }
//! Modify the bias of the layer.
arma::mat& Bias() { return bias; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the gradient.
OutputDataType const& Gradient() const { return gradient; }
//! Modify the gradient.
OutputDataType& Gradient() { return gradient; }
OutputType& Bias() { return bias; }
//! Get the input width.
size_t InputWidth() const { return inputWidth; }
const size_t& InputWidth() const { return inputWidth; }
//! Modify input the width.
size_t& InputWidth() { return inputWidth; }
//! Get the input height.
size_t InputHeight() const { return inputHeight; }
const size_t& InputHeight() const { return inputHeight; }
//! Modify the input height.
size_t& InputHeight() { return inputHeight; }
//! Get the output width.
size_t OutputWidth() const { return outputWidth; }
const size_t& OutputWidth() const { return outputWidth; }
//! Modify the output width.
size_t& OutputWidth() { return outputWidth; }
//! Get the output height.
size_t OutputHeight() const { return outputHeight; }
const size_t& OutputHeight() const { return outputHeight; }
//! Modify the output height.
size_t& OutputHeight() { return outputHeight; }
//! Get the input size.
size_t InputSize() const { return inSize; }
//! Get the output size.
size_t OutputSize() const { return outSize; }
//! Get the kernel width.
size_t KernelWidth() const { return kernelWidth; }
const size_t& KernelWidth() const { return kernelWidth; }
//! Modify the kernel width.
size_t& KernelWidth() { return kernelWidth; }
//! Get the kernel height.
size_t KernelHeight() const { return kernelHeight; }
const size_t& KernelHeight() const { return kernelHeight; }
//! Modify the kernel height.
size_t& KernelHeight() { return kernelHeight; }
//! Get the stride width.
size_t StrideWidth() const { return strideWidth; }
const size_t& StrideWidth() const { return strideWidth; }
//! Modify the stride width.
size_t& StrideWidth() { return strideWidth; }
//! Get the stride height.
size_t StrideHeight() const { return strideHeight; }
const size_t& StrideHeight() const { return strideHeight; }
//! Modify the stride height.
size_t& StrideHeight() { return strideHeight; }
//! Get the dilation rate on the X axis.
size_t DilationWidth() const { return dilationWidth; }
const size_t& DilationWidth() const { return dilationWidth; }
//! Modify the dilation rate on the X axis.
size_t& DilationWidth() { return dilationWidth; }
//! Get the dilation rate on the Y axis.
size_t DilationHeight() const { return dilationHeight; }
const size_t& DilationHeight() const { return dilationHeight; }
//! Modify the dilation rate on the Y axis.
size_t& DilationHeight() { return dilationHeight; }
//! Get the internal Padding layer.
ann::Padding<> const& Padding() const { return padding; }
PaddingType<InputType, OutputType> const& Padding() const { return padding; }
//! Modify the internal Padding layer.
ann::Padding<>& Padding() { return padding; }
PaddingType<InputType, OutputType>& Padding() { return padding; }
//! Get size of the weight matrix.
size_t WeightSize() const
@@ -358,13 +346,13 @@ class AtrousConvolution
size_t strideHeight;
//! Locally-stored weight object.
OutputDataType weights;
OutputType weights;
//! Locally-stored weight object.
arma::cube weight;
arma::Cube<typename OutputType::elem_type> weight;
//! Locally-stored bias term object.
arma::mat bias;
OutputType bias;
//! Locally-stored input width.
size_t inputWidth;
@@ -385,28 +373,19 @@ class AtrousConvolution
size_t dilationHeight;
//! Locally-stored transformed output parameter.
arma::cube outputTemp;
arma::Cube<typename OutputType::elem_type> outputTemp;
//! Locally-stored transformed padded input parameter.
arma::cube inputPaddedTemp;
arma::Cube<typename OutputType::elem_type> inputPaddedTemp;
//! Locally-stored transformed error parameter.
arma::cube gTemp;
arma::Cube<typename OutputType::elem_type> gTemp;
//! Locally-stored transformed gradient parameter.
arma::cube gradientTemp;
arma::Cube<typename OutputType::elem_type> gradientTemp;
//! Locally-stored padding layer.
ann::Padding<> padding;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored gradient object.
OutputDataType gradient;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
PaddingType<InputType, OutputType> padding;
}; // class AtrousConvolution
} // namespace ann
src/mlpack/methods/ann/layer/atrous_convolution_impl.hpp → src/mlpack/methods/ann/layer/not_adapted/atrous_convolution_impl.hpp
+73 -76
View File
@@ -23,15 +23,15 @@ template<
typename ForwardConvolutionRule,
typename BackwardConvolutionRule,
typename GradientConvolutionRule,
typename InputDataType,
typename OutputDataType
typename InputType,
typename OutputType
>
AtrousConvolution<
ForwardConvolutionRule,
BackwardConvolutionRule,
GradientConvolutionRule,
InputDataType,
OutputDataType
InputType,
OutputType
>::AtrousConvolution()
{
// Nothing to do here.
@@ -41,15 +41,15 @@ template<
typename ForwardConvolutionRule,
typename BackwardConvolutionRule,
typename GradientConvolutionRule,
typename InputDataType,
typename OutputDataType
typename InputType,
typename OutputType
>
AtrousConvolution<
ForwardConvolutionRule,
BackwardConvolutionRule,
GradientConvolutionRule,
InputDataType,
OutputDataType
InputType,
OutputType
>::AtrousConvolution(
const size_t inSize,
const size_t outSize,
@@ -86,15 +86,15 @@ template<
typename ForwardConvolutionRule,
typename BackwardConvolutionRule,
typename GradientConvolutionRule,
typename InputDataType,
typename OutputDataType
typename InputType,
typename OutputType
>
AtrousConvolution<
ForwardConvolutionRule,
BackwardConvolutionRule,
GradientConvolutionRule,
InputDataType,
OutputDataType
InputType,
OutputType
>::AtrousConvolution(
const size_t inSize,
const size_t outSize,
@@ -143,49 +143,49 @@ AtrousConvolution<
InitializeSamePadding(padWLeft, padWRight, padHTop, padHBottom);
}
padding = ann::Padding<>(padWLeft, padWRight, padHTop, padHBottom);
padding = PaddingType<InputType, OutputType>(padWLeft, padWRight, padHTop,
padHBottom);
}
template<
typename ForwardConvolutionRule,
typename BackwardConvolutionRule,
typename GradientConvolutionRule,
typename InputDataType,
typename OutputDataType
typename InputType,
typename OutputType
>
void AtrousConvolution<
ForwardConvolutionRule,
BackwardConvolutionRule,
GradientConvolutionRule,
InputDataType,
OutputDataType
>::Reset()
InputType,
OutputType
>::ResetWeights(typename OutputType::elem_type* weightsPtr)
{
weight = arma::cube(weights.memptr(), kernelWidth, kernelHeight,
outSize * inSize, false, false);
bias = arma::mat(weights.memptr() + weight.n_elem,
outSize, 1, false, false);
weight = arma::Cube<typename OutputType::elem_type>(weightsPtr, kernelWidth,
kernelHeight, outSize * inSize, false, true);
bias = OutputType(weightsPtr + weight.n_elem, outSize, 1, false, true);
}
template<
typename ForwardConvolutionRule,
typename BackwardConvolutionRule,
typename GradientConvolutionRule,
typename InputDataType,
typename OutputDataType
typename InputType,
typename OutputType
>
template<typename eT>
void AtrousConvolution<
ForwardConvolutionRule,
BackwardConvolutionRule,
GradientConvolutionRule,
InputDataType,
OutputDataType
>::Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output)
InputType,
OutputType
>::Forward(const InputType& input, OutputType& output)
{
batchSize = input.n_cols;
arma::cube inputTemp(const_cast<arma::Mat<eT>&>(input).memptr(),
inputWidth, inputHeight, inSize * batchSize, false, false);
arma::Cube<typename InputType::elem_type> inputTemp(
const_cast<InputType&>(input).memptr(), inputWidth, inputHeight, inSize *
batchSize, false, false);
if (padding.PadWLeft() != 0 || padding.PadWRight() != 0 ||
padding.PadHTop() != 0 || padding.PadHBottom() != 0)
@@ -207,8 +207,8 @@ void AtrousConvolution<
padding.PadHTop(), padding.PadHBottom(), dilationHeight);
output.set_size(wConv * hConv * outSize, batchSize);
outputTemp = arma::Cube<eT>(output.memptr(), wConv, hConv,
outSize * batchSize, false, false);
outputTemp = arma::Cube<typename OutputType::elem_type>(output.memptr(),
wConv, hConv, outSize * batchSize, false, false);
outputTemp.zeros();
for (size_t outMap = 0, outMapIdx = 0, batchCount = 0; outMap <
@@ -222,7 +222,7 @@ void AtrousConvolution<
for (size_t inMap = 0; inMap < inSize; inMap++, outMapIdx++)
{
arma::Mat<eT> convOutput;
OutputType convOutput;
if (padding.PadWLeft() != 0 || padding.PadWRight() != 0 ||
padding.PadHTop() != 0 || padding.PadHBottom() != 0)
@@ -252,25 +252,24 @@ template<
typename ForwardConvolutionRule,
typename BackwardConvolutionRule,
typename GradientConvolutionRule,
typename InputDataType,
typename OutputDataType
typename InputType,
typename OutputType
>
template<typename eT>
void AtrousConvolution<
ForwardConvolutionRule,
BackwardConvolutionRule,
GradientConvolutionRule,
InputDataType,
OutputDataType
>::Backward(
const arma::Mat<eT>& /* input */, const arma::Mat<eT>& gy, arma::Mat<eT>& g)
InputType,
OutputType
>::Backward(const InputType& /* input */, const OutputType& gy, OutputType& g)
{
arma::cube mappedError(((arma::Mat<eT>&) gy).memptr(), outputWidth,
outputHeight, outSize * batchSize, false, false);
arma::Cube<typename OutputType::elem_type> mappedError(
((OutputType&) gy).memptr(), outputWidth, outputHeight, outSize *
batchSize, false, false);
g.set_size(inputWidth * inputHeight * inSize, batchSize);
gTemp = arma::Cube<eT>(g.memptr(), inputWidth, inputHeight,
inSize * batchSize, false, false);
gTemp = arma::Cube<typename OutputType::elem_type>(g.memptr(), inputWidth,
inputHeight, inSize * batchSize, false, false);
gTemp.zeros();
for (size_t outMap = 0, outMapIdx = 0, batchCount = 0; outMap <
@@ -284,7 +283,7 @@ void AtrousConvolution<
for (size_t inMap = 0; inMap < inSize; inMap++, outMapIdx++)
{
arma::Mat<eT> output, rotatedFilter;
OutputType output, rotatedFilter;
Rotate180(weight.slice(outMapIdx), rotatedFilter);
BackwardConvolutionRule::Convolution(mappedError.slice(outMap),
@@ -311,29 +310,30 @@ template<
typename ForwardConvolutionRule,
typename BackwardConvolutionRule,
typename GradientConvolutionRule,
typename InputDataType,
typename OutputDataType
typename InputType,
typename OutputType
>
template<typename eT>
void AtrousConvolution<
ForwardConvolutionRule,
BackwardConvolutionRule,
GradientConvolutionRule,
InputDataType,
OutputDataType
InputType,
OutputType
>::Gradient(
const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient)
const InputType& input,
const OutputType& error,
OutputType& gradient)
{
arma::cube mappedError(((arma::Mat<eT>&) error).memptr(), outputWidth,
outputHeight, outSize * batchSize, false, false);
arma::cube inputTemp(const_cast<arma::Mat<eT>&>(input).memptr(),
inputWidth, inputHeight, inSize * batchSize, false, false);
arma::Cube<typename OutputType::elem_type> mappedError(
((OutputType&) error).memptr(), outputWidth, outputHeight, outSize *
batchSize, false, false);
arma::Cube<typename InputType::elem_type> inputTemp(
const_cast<InputType&>(input).memptr(), inputWidth, inputHeight,
inSize * batchSize, false, false);
gradient.set_size(weights.n_elem, 1);
gradientTemp = arma::Cube<eT>(gradient.memptr(), weight.n_rows,
weight.n_cols, weight.n_slices, false, false);
gradientTemp = arma::Cube<typename OutputType::elem_type>(gradient.memptr(),
weight.n_rows, weight.n_cols, weight.n_slices, false, false);
gradientTemp.zeros();
for (size_t outMap = 0, outMapIdx = 0, batchCount = 0; outMap <
@@ -347,7 +347,7 @@ void AtrousConvolution<
for (size_t inMap = 0; inMap < inSize; inMap++, outMapIdx++)
{
arma::Mat<eT> inputSlice;
InputType inputSlice;
if (padding.PadWLeft() != 0 || padding.PadWRight() != 0 ||
padding.PadHTop() != 0 || padding.PadHBottom() != 0)
{
@@ -358,9 +358,9 @@ void AtrousConvolution<
inputSlice = inputTemp.slice(inMap + batchCount * inSize);
}
arma::Mat<eT> deltaSlice = mappedError.slice(outMap);
OutputType deltaSlice = mappedError.slice(outMap);
arma::Mat<eT> output;
OutputType output;
GradientConvolutionRule::Convolution(inputSlice, deltaSlice,
output, strideWidth, strideHeight, 1, 1);
@@ -404,18 +404,20 @@ template<
typename ForwardConvolutionRule,
typename BackwardConvolutionRule,
typename GradientConvolutionRule,
typename InputDataType,
typename OutputDataType
typename InputType,
typename OutputType
>
template<typename Archive>
void AtrousConvolution<
ForwardConvolutionRule,
BackwardConvolutionRule,
GradientConvolutionRule,
InputDataType,
OutputDataType
InputType,
OutputType
>::serialize(Archive& ar, const uint32_t /* version */)
{
ar(cereal::base_class<Layer<InputType, OutputType>>(this));
ar(CEREAL_NVP(inSize));
ar(CEREAL_NVP(outSize));
ar(CEREAL_NVP(batchSize));
@@ -430,27 +432,22 @@ void AtrousConvolution<
ar(CEREAL_NVP(dilationWidth));
ar(CEREAL_NVP(dilationHeight));
ar(CEREAL_NVP(padding));
if (cereal::is_loading<Archive>())
{
weights.set_size((outSize * inSize * kernelWidth * kernelHeight) + outSize,
1);
}
ar(CEREAL_NVP(weights));
}
template<
typename ForwardConvolutionRule,
typename BackwardConvolutionRule,
typename GradientConvolutionRule,
typename InputDataType,
typename OutputDataType
typename InputType,
typename OutputType
>
void AtrousConvolution<
ForwardConvolutionRule,
BackwardConvolutionRule,
GradientConvolutionRule,
InputDataType,
OutputDataType
InputType,
OutputType
>::InitializeSamePadding(size_t& padWLeft,
size_t& padWRight,
size_t& padHTop,
src/mlpack/methods/ann/layer/batch_norm.hpp → src/mlpack/methods/ann/layer/not_adapted/batch_norm.hpp
+27 -65
View File
@@ -50,10 +50,10 @@ namespace ann /** Artificial Neural Network. */ {
* arma::sp_mat or arma::cube).
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
typename InputType = arma::mat,
typename OutputType = arma::mat
>
class BatchNorm
class BatchNorm : public Layer<InputType, OutputType>
{
public:
//! Create the BatchNorm object.
@@ -74,9 +74,9 @@ class BatchNorm
const double momentum = 0.1);
/**
* Reset the layer parameters
* Reset the layer parameters.
*/
void Reset();
void SetWeights(typename OutputType::elem_type* weightsPtr);
/**
* Forward pass of the Batch Normalization layer. Transforms the input data
@@ -86,8 +86,7 @@ class BatchNorm
* @param input Input data for the layer
* @param output Resulting output activations.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const InputType& input, OutputType& output);
/**
* Backward pass through the layer.
@@ -96,10 +95,9 @@ class BatchNorm
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& input,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g);
void Backward(const InputType& input,
const OutputType& gy,
OutputType& g);
/**
* Calculate the gradient using the output delta and the input activations.
@@ -108,45 +106,24 @@ class BatchNorm
* @param error The calculated error
* @param gradient The calculated gradient.
*/
template<typename eT>
void Gradient(const arma::Mat<eT>& input,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient);
void Gradient(const InputType& input,
const OutputType& error,
OutputType& gradient);
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
const OutputType& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the gradient.
OutputDataType const& Gradient() const { return gradient; }
//! Modify the gradient.
OutputDataType& Gradient() { return gradient; }
//! Get the value of deterministic parameter.
bool Deterministic() const { return deterministic; }
//! Modify the value of deterministic parameter.
bool& Deterministic() { return deterministic; }
OutputType& Parameters() { return weights; }
//! Get the mean over the training data.
OutputDataType const& TrainingMean() const { return runningMean; }
const OutputType& TrainingMean() const { return runningMean; }
//! Modify the mean over the training data.
OutputDataType& TrainingMean() { return runningMean; }
OutputType& TrainingMean() { return runningMean; }
//! Get the variance over the training data.
OutputDataType const& TrainingVariance() const { return runningVariance; }
const OutputType& TrainingVariance() const { return runningVariance; }
//! Modify the variance over the training data.
OutputDataType& TrainingVariance() { return runningVariance; }
OutputType& TrainingVariance() { return runningVariance; }
//! Get the number of input units / channels.
size_t InputSize() const { return size; }
@@ -187,25 +164,19 @@ class BatchNorm
bool loading;
//! Locally-stored scale parameter.
OutputDataType gamma;
OutputType gamma;
//! Locally-stored shift parameter.
OutputDataType beta;
OutputType beta;
//! Locally-stored mean object.
OutputDataType mean;
OutputType mean;
//! Locally-stored variance object.
OutputDataType variance;
OutputType variance;
//! Locally-stored parameters.
OutputDataType weights;
/**
* If true then mean and variance over the training set will be considered
* instead of being calculated over the batch.
*/
bool deterministic;
OutputType weights;
//! Locally-stored running mean/variance counter.
size_t count;
@@ -215,25 +186,16 @@ class BatchNorm
double averageFactor;
//! Locally-stored mean object.
OutputDataType runningMean;
OutputType runningMean;
//! Locally-stored variance object.
OutputDataType runningVariance;
//! Locally-stored gradient object.
OutputDataType gradient;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
OutputType runningVariance;
//! Locally-stored normalized input.
arma::cube normalized;
arma::Cube<typename InputType::elem_type> normalized;
//! Locally-stored zero mean input.
arma::cube inputMean;
arma::Cube<typename InputType::elem_type> inputMean;
}; // class BatchNorm
} // namespace ann
src/mlpack/methods/ann/layer/batch_norm_impl.hpp → src/mlpack/methods/ann/layer/not_adapted/batch_norm_impl.hpp
+64 -58
View File
@@ -21,22 +21,21 @@
namespace mlpack {
namespace ann { /** Artificial Neural Network. */
template<typename InputDataType, typename OutputDataType>
BatchNorm<InputDataType, OutputDataType>::BatchNorm() :
template<typename InputType, typename OutputType>
BatchNorm<InputType, OutputType>::BatchNorm() :
size(0),
eps(1e-8),
average(true),
momentum(0.0),
loading(false),
deterministic(false),
count(0),
averageFactor(0.0)
{
// Nothing to do here.
}
template <typename InputDataType, typename OutputDataType>
BatchNorm<InputDataType, OutputDataType>::BatchNorm(
template <typename InputType, typename OutputType>
BatchNorm<InputType, OutputType>::BatchNorm(
const size_t size,
const double eps,
const bool average,
@@ -46,7 +45,6 @@ BatchNorm<InputDataType, OutputDataType>::BatchNorm(
average(average),
momentum(momentum),
loading(false),
deterministic(false),
count(0),
averageFactor(0.0)
{
@@ -55,13 +53,14 @@ BatchNorm<InputDataType, OutputDataType>::BatchNorm(
runningVariance.ones(size, 1);
}
template<typename InputDataType, typename OutputDataType>
void BatchNorm<InputDataType, OutputDataType>::Reset()
template<typename InputType, typename OutputType>
void BatchNorm<InputType, OutputType>::SetWeights(
typename OutputType::elem_type* weightsPtr)
{
// Gamma acts as the scaling parameters for the normalized output.
gamma = arma::mat(weights.memptr(), size, 1, false, false);
gamma = OutputType(weightsPtr, size, 1, false, false);
// Beta acts as the shifting parameters for the normalized output.
beta = arma::mat(weights.memptr() + gamma.n_elem, size, 1, false, false);
beta = OutputType(weightsPtr + gamma.n_elem, size, 1, false, false);
if (!loading)
{
@@ -69,18 +68,16 @@ void BatchNorm<InputDataType, OutputDataType>::Reset()
beta.fill(0.0);
}
deterministic = false;
loading = false;
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void BatchNorm<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input,
arma::Mat<eT>& output)
template<typename InputType, typename OutputType>
void BatchNorm<InputType, OutputType>::Forward(
const InputType& input,
OutputType& output)
{
Log::Assert(input.n_rows % size == 0, "Input features must be divisible \
by feature maps.");
Log::Assert(input.n_rows % size == 0, "Input features must be divisible "
"by feature maps.");
const size_t batchSize = input.n_cols;
const size_t inputSize = input.n_rows / size;
@@ -89,7 +86,7 @@ void BatchNorm<InputDataType, OutputDataType>::Forward(
output.set_size(arma::size(input));
// We will calculate minibatch norm on each channel / feature map.
if (!deterministic)
if (this->training)
{
// Check only during training, batch-size can be one during inference.
if (batchSize == 1 && inputSize == 1)
@@ -101,12 +98,14 @@ void BatchNorm<InputDataType, OutputDataType>::Forward(
// Input corresponds to output from convolution layer.
// Use a cube for simplicity.
arma::cube inputTemp(const_cast<arma::Mat<eT>&>(input).memptr(),
inputSize, size, batchSize, false, false);
arma::Cube<typename InputType::elem_type> inputTemp(
const_cast<InputType&>(input).memptr(), inputSize, size, batchSize,
false, false);
// Initialize output to same size and values for convenience.
arma::cube outputTemp(const_cast<arma::Mat<eT>&>(output).memptr(),
inputSize, size, batchSize, false, false);
arma::Cube<typename OutputType::elem_type> outputTemp(
const_cast<OutputType&>(output).memptr(), inputSize, size, batchSize,
false, false);
outputTemp = inputTemp;
// Calculate mean and variance over all channels.
@@ -152,8 +151,9 @@ void BatchNorm<InputDataType, OutputDataType>::Forward(
{
// Normalize the input and scale and shift the output.
output = input;
arma::cube outputTemp(const_cast<arma::Mat<eT>&>(output).memptr(),
input.n_rows / size, size, batchSize, false, false);
arma::Cube<typename OutputType::elem_type> outputTemp(
const_cast<OutputType&>(output).memptr(), input.n_rows / size, size,
batchSize, false, false);
outputTemp.each_slice() -= arma::repmat(runningMean.t(),
input.n_rows / size, 1);
@@ -166,28 +166,29 @@ void BatchNorm<InputDataType, OutputDataType>::Forward(
}
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void BatchNorm<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& input,
const arma::Mat<eT>& gy,
arma::Mat<eT>& g)
template<typename InputType, typename OutputType>
void BatchNorm<InputType, OutputType>::Backward(
const InputType& input,
const OutputType& gy,
OutputType& g)
{
const arma::mat stdInv = 1.0 / arma::sqrt(variance + eps);
g.set_size(arma::size(input));
arma::cube gyTemp(const_cast<arma::Mat<eT>&>(gy).memptr(),
input.n_rows / size, size, input.n_cols, false, false);
arma::cube gTemp(const_cast<arma::Mat<eT>&>(g).memptr(),
input.n_rows / size, size, input.n_cols, false, false);
arma::Cube<typename OutputType::elem_type> gyTemp(
const_cast<OutputType&>(gy).memptr(), input.n_rows / size, size,
input.n_cols, false, false);
arma::Cube<typename OutputType::elem_type> gTemp(
const_cast<OutputType&>(g).memptr(), input.n_rows / size, size,
input.n_cols, false, false);
// Step 1: dl / dxhat.
arma::cube norm = gyTemp.each_slice() % arma::repmat(gamma.t(),
input.n_rows / size, 1);
arma::Cube<typename OutputType::elem_type> norm =
gyTemp.each_slice() % arma::repmat(gamma.t(), input.n_rows / size, 1);
// Step 2: sum dl / dxhat * (x - mu) * -0.5 * stdInv^3.
arma::mat temp = arma::sum(norm % inputMean, 2);
arma::mat vars = temp % arma::repmat(arma::pow(stdInv, 3),
OutputType temp = arma::sum(norm % inputMean, 2);
OutputType vars = temp % arma::repmat(arma::pow(stdInv, 3),
input.n_rows / size, 1) * -0.5;
// Step 3: dl / dxhat * 1 / stdInv + variance * 2 * (x - mu) / m +
@@ -198,25 +199,25 @@ void BatchNorm<InputDataType, OutputDataType>::Backward(
// Step 4: sum (dl / dxhat * -1 / stdInv) + variance *
// (sum -2 * (x - mu)) / m.
arma::mat normTemp = arma::sum(norm.each_slice() %
OutputType normTemp = arma::sum(norm.each_slice() %
arma::repmat(-stdInv, input.n_rows / size, 1) , 2) /
input.n_cols;
gTemp.each_slice() += normTemp;
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void BatchNorm<InputDataType, OutputDataType>::Gradient(
const arma::Mat<eT>& /* input */,
const arma::Mat<eT>& error,
arma::Mat<eT>& gradient)
template<typename InputType, typename OutputType>
void BatchNorm<InputType, OutputType>::Gradient(
const InputType& /* input */,
const OutputType& error,
OutputType& gradient)
{
gradient.set_size(size + size, 1);
arma::cube errorTemp(const_cast<arma::Mat<eT>&>(error).memptr(),
error.n_rows / size, size, error.n_cols, false, false);
arma::Cube<typename OutputType::elem_type> errorTemp(
const_cast<OutputType&>(error).memptr(), error.n_rows / size, size,
error.n_cols, false, false);
// Step 5: dl / dy * xhat.
arma::mat temp = arma::sum(arma::sum(normalized % errorTemp, 0), 2);
OutputType temp = arma::sum(arma::sum(normalized % errorTemp, 0), 2);
gradient.submat(0, 0, gamma.n_elem - 1, 0) = temp.t();
// Step 6: dl / dy.
@@ -224,22 +225,27 @@ void BatchNorm<InputDataType, OutputDataType>::Gradient(
gradient.submat(gamma.n_elem, 0, gradient.n_elem - 1, 0) = temp.t();
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
template<typename Archive>
void BatchNorm<InputDataType, OutputDataType>::serialize(
void BatchNorm<InputType, OutputType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(cereal::base_class<Layer<InputType, OutputType>>(this));
ar(CEREAL_NVP(size));
if (cereal::is_loading<Archive>())
{
weights.set_size(size + size, 1);
loading = true;
}
ar(CEREAL_NVP(eps));
ar(CEREAL_NVP(gamma));
ar(CEREAL_NVP(beta));
ar(CEREAL_NVP(weights));
if (Archive::is_loading::value)
{
// Gamma acts as the scaling parameters for the normalized output.
gamma = arma::mat(weights.memptr(), size, 1, false, false);
// Beta acts as the shifting parameters for the normalized output.
beta = arma::mat(weights.memptr() + gamma.n_elem, size, 1, false, false);
}
ar(CEREAL_NVP(count));
ar(CEREAL_NVP(averageFactor));
ar(CEREAL_NVP(momentum));
src/mlpack/methods/ann/layer/bicubic_interpolation.hpp → src/mlpack/methods/ann/layer/not_adapted/bicubic_interpolation.hpp
View File
src/mlpack/methods/ann/layer/bicubic_interpolation_impl.hpp → src/mlpack/methods/ann/layer/not_adapted/bicubic_interpolation_impl.hpp
View File
src/mlpack/methods/ann/layer/bilinear_interpolation.hpp → src/mlpack/methods/ann/layer/not_adapted/bilinear_interpolation.hpp
+32 -64
View File
@@ -1,3 +1,4 @@
// Temporarily drop.
/**
* @file methods/ann/layer/bilinear_interpolation.hpp
* @author Kris Singh
@@ -27,35 +28,30 @@ namespace ann /** Artificial Neural Network. */ {
* different known points in the grid. This way, we represent any arbitrary
* point, present within the grid, as a function of those four points.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* @tparam InputType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* @tparam OutputType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
typename InputType = arma::mat,
typename OutputType = arma::mat
>
class BilinearInterpolation
class BilinearInterpolationType : public Layer<InputType, OutputType>
{
public:
//! Create the Bilinear Interpolation object.
BilinearInterpolation();
//! Create the BilinearInterpolationType object.
BilinearInterpolationType();
/**
* The constructor for the Bilinear Interpolation.
* The constructor for the Bilinear Interpolation. The input size will be set
* by the given input when the layer is used.
*
* @param inRowSize Number of input rows.
* @param inColSize Number of input columns.
* @param outRowSize Number of output rows.
* @param outColSize Number of output columns.
* @param depth Number of input slices.
*/
BilinearInterpolation(const size_t inRowSize,
const size_t inColSize,
const size_t outRowSize,
const size_t outColSize,
const size_t depth);
BilinearInterpolationType(const size_t outRowSize,
const size_t outColSize);
/**
* Forward pass through the layer. The layer interpolates
@@ -64,8 +60,7 @@ class BilinearInterpolation
* @param input The input matrix.
* @param output The resulting interpolated output matrix.
*/
template<typename eT>
void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output);
void Forward(const InputType& input, OutputType& output);
/**
* Ordinary feed backward pass of a neural network, calculating the function
@@ -78,30 +73,22 @@ class BilinearInterpolation
* @param gradient The computed backward gradient.
* @param output The resulting down-sampled output.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& /*input*/,
const arma::Mat<eT>& gradient,
arma::Mat<eT>& output);
void Backward(const InputType& /*input*/,
const OutputType& gradient,
OutputType& output);
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the row size of the input.
size_t const& InRowSize() const { return inRowSize; }
//! Modify the row size of the input.
size_t& InRowSize() { return inRowSize; }
//! Get the column size of the input.
size_t const& InColSize() const { return inColSize; }
//! Modify the column size of the input.
size_t& InColSize() { return inColSize; }
const std::vector<size_t>& OutputDimensions() const
{
std::vector<size_t> result(this->inputDimensions.size(), 0);
result[0] = outRowSize;
result[1] = outColSize;
if (result.size() > 2)
{
for (size_t i = 0; i < result.size(); ++i)
result[i] = this->inputDimensions[i];
}
return result;
}
//! Get the row size of the output.
size_t const& OutRowSize() const { return outRowSize; }
@@ -113,17 +100,6 @@ class BilinearInterpolation
//! Modify the column size of the output.
size_t& OutColSize() { return outColSize; }
//! Get the depth of the input.
size_t const& InDepth() const { return depth; }
//! Modify the depth of the input.
size_t& InDepth() { return depth; }
//! Get the shape of the input.
size_t InputShape() const
{
return inRowSize;
}
/**
* Serialize the layer.
*/
@@ -131,24 +107,16 @@ class BilinearInterpolation
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Locally stored row size of the input.
size_t inRowSize;
//! Locally stored column size of the input.
size_t inColSize;
//! Locally stored row size of the output.
size_t outRowSize;
//! Locally stored column size of the input.
size_t outColSize;
//! Locally stored depth of the input.
size_t depth;
//! Locally stored number of input points.
size_t batchSize;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
}; // class BilinearInterpolation
// Standard BilinearInterpolation layer.
typedef BilinearInterpolationType<arma::mat, arma::mat> BilinearInterpolation;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/bilinear_interpolation_impl.hpp → src/mlpack/methods/ann/layer/not_adapted/bilinear_interpolation_impl.hpp
+55 -72
View File
@@ -19,69 +19,54 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
BilinearInterpolation<InputDataType, OutputDataType>::
BilinearInterpolation():
inRowSize(0),
inColSize(0),
template<typename InputType, typename OutputType>
BilinearInterpolationType<InputType, OutputType>::
BilinearInterpolationType():
outRowSize(0),
outColSize(0),
depth(0),
batchSize(0)
outColSize(0)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
BilinearInterpolation<InputDataType, OutputDataType>::
BilinearInterpolation(
const size_t inRowSize,
const size_t inColSize,
const size_t outRowSize,
const size_t outColSize,
const size_t depth):
inRowSize(inRowSize),
inColSize(inColSize),
template<typename InputType, typename OutputType>
BilinearInterpolationType<InputType, OutputType>::
BilinearInterpolationType(const size_t outRowSize,
const size_t outColSize) :
outRowSize(outRowSize),
outColSize(outColSize),
depth(depth),
batchSize(0)
outColSize(outColSize)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void BilinearInterpolation<InputDataType, OutputDataType>::Forward(
const arma::Mat<eT>& input, arma::Mat<eT>& output)
template<typename InputType, typename OutputType>
void BilinearInterpolationType<InputType, OutputType>::Forward(
const InputType& input, OutputType& output)
{
batchSize = input.n_cols;
if (output.is_empty())
output.set_size(outRowSize * outColSize * depth, batchSize);
else
{
assert(output.n_rows == outRowSize * outColSize * depth);
assert(output.n_cols == batchSize);
}
const size_t batchSize = input.n_cols;
const size_t depth = this->inputDimensions.size() <= 2 ? 1 :
std::accumulate(this->inputDimensions.begin() + 2, this->inputDimensions.end(), 0);
assert(inRowSize >= 2);
assert(inColSize >= 2);
assert(output.n_rows == outRowSize * outColSize * depth);
assert(output.n_cols == batchSize);
arma::cube inputAsCube(const_cast<arma::Mat<eT>&>(input).memptr(),
inRowSize, inColSize, depth * batchSize, false, false);
arma::cube outputAsCube(output.memptr(), outRowSize, outColSize,
depth * batchSize, false, true);
assert(this->inputDimensions[0] >= 2);
assert(this->inputDimensions[1] >= 2);
double scaleRow = (double) inRowSize / (double) outRowSize;
double scaleCol = (double) inColSize / (double) outColSize;
arma::Cube<typename InputType::elem_type> inputAsCube(
const_cast<InputType&>(input).memptr(), this->inputDimensions[0],
this->inputDimensions[1], depth * batchSize, false, false);
arma::Cube<typename OutputType::elem_type> outputAsCube(
output.memptr(), outRowSize, outColSize, depth * batchSize, false, true);
double scaleRow = (double) this->inputDimensions[0] / (double) outRowSize;
double scaleCol = (double) this->inputDimensions[1] / (double) outColSize;
arma::mat22 coeffs;
for (size_t i = 0; i < outRowSize; ++i)
{
size_t rOrigin = (size_t) std::floor(i * scaleRow);
if (rOrigin > inRowSize - 2)
rOrigin = inRowSize - 2;
if (rOrigin > this->inputDimensions[0] - 2)
rOrigin = this->inputDimensions[0] - 2;
// Scaled distance of the interpolated point from the topmost row.
double deltaR = i * scaleRow - rOrigin;
@@ -91,8 +76,8 @@ void BilinearInterpolation<InputDataType, OutputDataType>::Forward(
{
// Scaled distance of the interpolated point from the leftmost column.
size_t cOrigin = (size_t) std::floor(j * scaleCol);
if (cOrigin > inColSize - 2)
cOrigin = inColSize - 2;
if (cOrigin > this->inputDimensions[1] - 2)
cOrigin = this->inputDimensions[1] - 2;
double deltaC = j * scaleCol - cOrigin;
if (deltaC > 1)
@@ -111,28 +96,27 @@ void BilinearInterpolation<InputDataType, OutputDataType>::Forward(
}
}
template<typename InputDataType, typename OutputDataType>
template<typename eT>
void BilinearInterpolation<InputDataType, OutputDataType>::Backward(
const arma::Mat<eT>& /*input*/,
const arma::Mat<eT>& gradient,
arma::Mat<eT>& output)
template<typename InputType, typename OutputType>
void BilinearInterpolationType<InputType, OutputType>::Backward(
const InputType& /*input*/,
const OutputType& gradient,
OutputType& output)
{
if (output.is_empty())
output.set_size(inRowSize * inColSize * depth, batchSize);
else
{
assert(output.n_rows == inRowSize * inColSize * depth);
assert(output.n_cols == batchSize);
}
const size_t batchSize = output.n_cols;
const size_t depth = this->inputDimensions.size() <= 2 ? 1 :
std::accumulate(this->inputDimensions.begin() + 2, this->inputDimensions.end(), 0);
assert(output.n_rows == this->inputDimensions[0] * this->inputDimensions[1] * depth);
assert(outRowSize >= 2);
assert(outColSize >= 2);
arma::cube gradientAsCube(((arma::Mat<eT>&) gradient).memptr(), outRowSize,
outColSize, depth * batchSize, false, false);
arma::cube outputAsCube(output.memptr(), inRowSize, inColSize,
depth * batchSize, false, true);
arma::Cube<typename OutputType::elem_type> gradientAsCube(
((OutputType&) gradient).memptr(), outRowSize, outColSize, depth *
batchSize, false, false);
arma::Cube<typename OutputType::elem_type> outputAsCube(
output.memptr(), this->inputDimensions[0], this->inputDimensions[1], depth * batchSize,
false, true);
if (gradient.n_elem == output.n_elem)
{
@@ -140,17 +124,17 @@ void BilinearInterpolation<InputDataType, OutputDataType>::Backward(
}
else
{
double scaleRow = (double)(outRowSize) / inRowSize;
double scaleCol = (double)(outColSize) / inColSize;
double scaleRow = (double)(outRowSize) / this->inputDimensions[0];
double scaleCol = (double)(outColSize) / this->inputDimensions[1];
arma::mat22 coeffs;
for (size_t i = 0; i < inRowSize; ++i)
for (size_t i = 0; i < this->inputDimensions[0]; ++i)
{
size_t rOrigin = (size_t) std::floor(i * scaleRow);
if (rOrigin > outRowSize - 2)
rOrigin = outRowSize - 2;
double deltaR = i * scaleRow - rOrigin;
for (size_t j = 0; j < inColSize; ++j)
for (size_t j = 0; j < this->inputDimensions[1]; ++j)
{
size_t cOrigin = (size_t) std::floor(j * scaleCol);
@@ -173,16 +157,15 @@ void BilinearInterpolation<InputDataType, OutputDataType>::Backward(
}
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
template<typename Archive>
void BilinearInterpolation<InputDataType, OutputDataType>::serialize(
void BilinearInterpolationType<InputType, OutputType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(CEREAL_NVP(inRowSize));
ar(CEREAL_NVP(inColSize));
ar(cereal::base_class<Layer<InputType, OutputType>>(this));
ar(CEREAL_NVP(outRowSize));
ar(CEREAL_NVP(outColSize));
ar(CEREAL_NVP(depth));
}
} // namespace ann
src/mlpack/methods/ann/layer/c_relu.hpp → src/mlpack/methods/ann/layer/not_adapted/c_relu.hpp
+21 -39
View File
@@ -14,10 +14,12 @@
#include <mlpack/prereqs.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
/**
*
* A concatenated ReLU has two outputs, one ReLU and one negative ReLU,
* concatenated together. In other words, for positive x it produces [x, 0],
* and for negative x it produces [0, x]. Because it has two outputs,
@@ -38,22 +40,21 @@ namespace ann /** Artificial Neural Network. */ {
* }
* @endcode
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam InputType The type of the layer's inputs. The layer automatically
* cast inputs to this type (Default: arma::mat).
* @tparam OutputType The type of the computation which also causes the output
* to also be in this type. The type also allows the computation and weight
* type to differ from the input type (Default: arma::mat).
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class CReLU
template<typename InputType = arma::mat, typename OutputType = arma::mat>
class CReLUType : public Layer<InputType, OutputType>
{
public:
/**
* Create the CReLU object.
*/
CReLU();
//! Create the CReLU object.
CReLUType();
//! Clone the CReLUType object. This handles polymorphism correctly.
CReLUType* Clone() const { return new CReLUType(*this); }
/**
* Ordinary feed forward pass of a neural network, evaluating the function
@@ -63,7 +64,6 @@ class CReLU
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename InputType, typename OutputType>
void Forward(const InputType& input, OutputType& output);
/**
@@ -75,35 +75,17 @@ class CReLU
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename DataType>
void Backward(const DataType& input, const DataType& gy, DataType& g);
void Backward(const InputType& input, const OutputType& gy, OutputType& g);
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get size of weights.
size_t WeightSize() const { return 0; }
/**
* Serialize the layer.
*/
//! Serialize the layer.
template<typename Archive>
void serialize(Archive& /* ar */, const uint32_t /* version */);
}; // class CReLUType
private:
//! Locally-stored delta object.
OutputDataType delta;
// Convenience typedefs.
//! Locally-stored output parameter object.
OutputDataType outputParameter;
}; // class CReLU
// Standard CReLU layer.
typedef CReLUType<arma::mat, arma::mat> CReLU;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/c_relu_impl.hpp → src/mlpack/methods/ann/layer/not_adapted/c_relu_impl.hpp
+11 -14
View File
@@ -18,39 +18,36 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
CReLU<InputDataType, OutputDataType>::CReLU()
template<typename InputType, typename OutputType>
CReLUType<InputType, OutputType>::CReLUType()
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
void CReLU<InputDataType, OutputDataType>::Forward(
void CReLUType<InputType, OutputType>::Forward(
const InputType& input, OutputType& output)
{
output = arma::join_cols(arma::max(input, 0.0 * input), arma::max(
(-1 * input), 0.0 * input));
}
template<typename InputDataType, typename OutputDataType>
template<typename DataType>
void CReLU<InputDataType, OutputDataType>::Backward(
const DataType& input, const DataType& gy, DataType& g)
template<typename InputType, typename OutputType>
void CReLUType<InputType, OutputType>::Backward(
const InputType& input, const OutputType& gy, OutputType& g)
{
DataType temp;
temp = gy % (input >= 0.0);
OutputType temp = gy % (input >= 0.0);
g = temp.rows(0, (input.n_rows / 2 - 1)) - temp.rows(input.n_rows / 2,
(input.n_rows - 1));
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
template<typename Archive>
void CReLU<InputDataType, OutputDataType>::serialize(
Archive& /* ar */,
void CReLUType<InputType, OutputType>::serialize(
Archive& ar,
const uint32_t /* version */)
{
// Nothing to do here.
ar(cereal::base_class<Layer<InputType, OutputType>>(this));
}
} // namespace ann
src/mlpack/methods/ann/layer/celu.hpp → src/mlpack/methods/ann/layer/not_adapted/celu.hpp
+22 -45
View File
@@ -25,6 +25,8 @@
#include <mlpack/prereqs.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -46,18 +48,16 @@ namespace ann /** Artificial Neural Network. */ {
* \right.
* @f}
*
* In the deterministic mode, there is no computation of the derivative.
* When not in training mode, there is no computation of the derivative.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam InputType The type of the layer's inputs. The layer automatically
* cast inputs to this type (Default: arma::mat).
* @tparam OutputType The type of the computation which also causes the output
* to also be in this type. The type also allows the computation and weight
* type to differ from the input type (Default: arma::mat).
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class CELU
template<typename InputType = arma::mat, typename OutputType = arma::mat>
class CELUType : public Layer<InputType, OutputType>
{
public:
/**
@@ -67,7 +67,10 @@ class CELU
*
* @param alpha Scale parameter for the negative factor (default = 1.0).
*/
CELU(const double alpha = 1.0);
CELUType(const double alpha = 1.0);
//! Clone the CELUType object. This handles polymorphism correctly.
CELUType* Clone() const { return new CELUType(*this); }
/**
* Ordinary feed forward pass of a neural network, evaluating the function
@@ -76,7 +79,6 @@ class CELU
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename InputType, typename OutputType>
void Forward(const InputType& input, OutputType& output);
/**
@@ -88,54 +90,29 @@ class CELU
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename DataType>
void Backward(const DataType& input, const DataType& gy, DataType& g);
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
void Backward(const InputType& input, const OutputType& gy, OutputType& g);
//! Get the non zero gradient.
double const& Alpha() const { return alpha; }
//! Modify the non zero gradient.
double& Alpha() { return alpha; }
//! Get the value of deterministic parameter.
bool Deterministic() const { return deterministic; }
//! Modify the value of deterministic parameter.
bool& Deterministic() { return deterministic; }
//! Get size of weights.
size_t WeightSize() { return 0; }
/**
* Serialize the layer.
*/
//! Serialize the layer.
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
//! Locally stored first derivative of the activation function.
arma::mat derivative;
OutputType derivative;
//! CELU Hyperparameter (alpha > 0).
double alpha;
}; // class CELUType
//! If true the derivative computation is disabled, see notes above.
bool deterministic;
}; // class CELU
// Convenience typedefs.
// Standard CELU layer.
typedef CELUType<arma::mat, arma::mat> CELU;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/celu_impl.hpp → src/mlpack/methods/ann/layer/not_adapted/celu_impl.hpp
+16 -15
View File
@@ -18,10 +18,9 @@
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputDataType, typename OutputDataType>
CELU<InputDataType, OutputDataType>::CELU(const double alpha) :
alpha(alpha),
deterministic(false)
template<typename InputType, typename OutputType>
CELUType<InputType, OutputType>::CELUType(const double alpha) :
alpha(alpha)
{
if (alpha == 0)
{
@@ -30,19 +29,18 @@ CELU<InputDataType, OutputDataType>::CELU(const double alpha) :
}
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
void CELU<InputDataType, OutputDataType>::Forward(
void CELUType<InputType, OutputType>::Forward(
const InputType& input, OutputType& output)
{
output = arma::ones<OutputDataType>(arma::size(input));
output = arma::ones<OutputType>(arma::size(input));
for (size_t i = 0; i < input.n_elem; ++i)
{
output(i) = (input(i) >= 0) ? input(i) : alpha *
(std::exp(input(i) / alpha) - 1);
(std::exp(input(i) / alpha) - 1);
}
if (!deterministic)
if (this->training)
{
derivative.set_size(arma::size(input));
for (size_t i = 0; i < input.n_elem; ++i)
@@ -53,21 +51,24 @@ void CELU<InputDataType, OutputDataType>::Forward(
}
}
template<typename InputDataType, typename OutputDataType>
template<typename DataType>
void CELU<InputDataType, OutputDataType>::Backward(
const DataType& /* input */, const DataType& gy, DataType& g)
template<typename InputType, typename OutputType>
void CELUType<InputType, OutputType>::Backward(
const InputType& /* input */, const OutputType& gy, OutputType& g)
{
g = gy % derivative;
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
template<typename Archive>
void CELU<InputDataType, OutputDataType>::serialize(
void CELUType<InputType, OutputType>::serialize(
Archive& ar,
const uint32_t /* version */)
{
ar(cereal::base_class<Layer<InputType, OutputType>>(this));
ar(CEREAL_NVP(alpha));
if (Archive::is_loading::value)
derivative.clear();
}
} // namespace ann
src/mlpack/methods/ann/layer/channel_shuffle.hpp → src/mlpack/methods/ann/layer/not_adapted/channel_shuffle.hpp
View File
src/mlpack/methods/ann/layer/channel_shuffle_impl.hpp → src/mlpack/methods/ann/layer/not_adapted/channel_shuffle_impl.hpp
View File
src/mlpack/methods/ann/layer/not_adapted/concat.hpp
+222
View File
@@ -0,0 +1,222 @@
/**
* @file methods/ann/layer/concat.hpp
* @author Marcus Edel
* @author Mehul Kumar Nirala
*
* Definition of the Concat class, which acts as a concatenation container.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_CONCAT_HPP
#define MLPACK_METHODS_ANN_LAYER_CONCAT_HPP
#include <mlpack/prereqs.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
/**
* Implementation of the Concat class. The Concat class works as a
* feed-forward fully connected network container which plugs various layers
* together.
*
* @tparam InputType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
*/
template <
typename InputType = arma::mat,
typename OutputType = arma::mat
>
class ConcatType : public MultiLayer<InputType, OutputType>
{
public:
/**
* Create the Concat object using the specified parameters.
*
* @param run Call the Forward/Backward method before the output is merged.
*/
ConcatType(const bool run = true);
/**
* Create the Concat object, specifying a particular axis on which the layer
* outputs should be concatenated.
*
* @param axis Concat axis.
* @param run Call the Forward/Backward method before the output is merged.
*/
ConcatType(const size_t axis, const bool run = true);
/**
* Destroy the layers held by the model.
*/
~ConcatType();
//! Clone the ConcatType object. This handles polymorphism correctly.
ConcatType* Clone() const { return new ConcatType(*this); }
/**
* Ordinary feed forward pass of a neural network, evaluating the function
* f(x) by propagating the activity forward through f.
*
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
void Forward(const InputType& input, OutputType& output);
/**
* Ordinary feed backward pass of a neural network, using 3rd-order tensors as
* input, calculating the function f(x) by propagating x backwards through f.
* Using the results from the feed forward pass.
*
* @param * (input) The propagated input activation.
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
void Backward(const InputType& /* input */,
const OutputType& gy,
OutputType& g);
/**
* This is the overload of Backward() that runs only a specific layer with
* the given input.
*
* @param * (input) The propagated input activation.
* @param gy The backpropagated error.
* @param g The calculated gradient.
* @param index The index of the layer to run.
*/
void Backward(const InputType& /* input */,
const OutputType& gy,
OutputType& g,
const size_t index);
/**
* Calculate the gradient using the output delta and the input activation.
*
* @param input The input parameter used for calculating the gradient.
* @param error The calculated error.
* @param gradient The calculated gradient.
*/
void Gradient(const InputType& /* input */,
const OutputType& error,
OutputType& /* gradient */);
/**
* This is the overload of Gradient() that runs a specific layer with the
* given input.
*
* @param input The input parameter used for calculating the gradient.
* @param error The calculated error.
* @param gradient The calculated gradient.
* @param The index of the layer to run.
*/
void Gradient(const InputType& input,
const OutputType& error,
OutputType& gradient,
const size_t index);
//! Get the value of run parameter.
bool Run() const { return run; }
//! Modify the value of run parameter.
bool& Run() { return run; }
//! Get the axis of concatenation.
const size_t& ConcatAxis() const { return axis; }
//! Get the size of the weight matrix.
size_t WeightSize() const { return 0; }
void ComputeOutputDimensions()
{
// The input is sent to every layer.
for (size_t i = 0; i < network.size(); ++i)
{
network[i]->InputDimensions() = this->inputDimensions;
network[i]->ComputeOutputDimensions();
}
// If the user did not specify an axis, we will use the last one.
// Otherwise, we must sanity check to ensure that the axis we are
// concatenating along is valid.
if (!useAxis)
{
axis = this->inputDimensions.size() - 1;
}
else if (axis >= this->inputDimensions.size())
{
std::ostringstream oss;
oss << "Concat::ComputeOutputDimensions(): cannot concatenate outputs "
<< "along axis " << axis << " when input only has "
<< this->inputDimensions.size() << " axes!";
throw std::invalid_argument(oss.str());
}
// Now, we concatenate the output along a specific axis.
this->outputDimensions = std::vector<size_t>(this->inputDimensions.size(),
0);
for (size_t i = 0; i < this->inputDimensions.size(); ++i)
{
if (i == axis)
{
// Accumulate output size along this axis for each layer output.
for (size_t n = 0; n < this->network.size(); ++n)
{
this->outputDimensions[i] += this->network[n]->OutputDimensions()[i];
}
}
else
{
// Ensure that the output size is the same along this axis.
const size_t axisDim = this->network[0]->OutputDimensions()[i];
for (size_t n = 1; n < this->network.size(); ++n)
{
const size_t axisDim2 = this->network[n]->OutputDimensions()[i];
if (axisDim != axisDim2)
{
std::ostringstream oss;
oss << "Concat::ComputeOutputDimensions(): cannot concatenate "
<< "outputs along axis " << axis << "; held layer " << n
<< " has output size " << axisDim2 << " along axis " << i
<< ", but the first held layer has output size " << axisDim
<< "! All layers must have identical output size in any "
<< "axis other than the concatenated axis.";
throw std::invalid_argument(oss.str());
}
}
this->outputDimensions[i] = axisDim;
}
}
}
/**
* Serialize the layer
*/
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Parameter which indicates the axis of concatenation.
size_t axis;
//! Parameter which indicates whether to use the axis of concatenation.
bool useAxis;
}; // class ConcatType.
// Standard Concat layer.
typedef ConcatType<arma::mat, arma::mat> Concat;
} // namespace ann
} // namespace mlpack
// Include implementation.
#include "concat_impl.hpp"
#endif
src/mlpack/methods/ann/layer/not_adapted/concat_impl.hpp
+267
View File
@@ -0,0 +1,267 @@
/**
* @file methods/ann/layer/concat_impl.hpp
* @author Marcus Edel
* @author Mehul Kumar Nirala
*
* Implementation of the Concat class, which acts as a concatenation contain.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_CONCAT_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_CONCAT_IMPL_HPP
// In case it hasn't yet been included.
#include "concat.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputType, typename OutputType>
ConcatType<InputType, OutputType>::ConcatType(
const bool run) :
axis(0),
useAxis(false)
{
// Nothing to do.
}
template<typename InputType, typename OutputType>
ConcatType<InputType, OutputType>::ConcatType(
const size_t axis,
const bool run) :
axis(axis),
useAxis(true)
{
// Nothing to do.
}
template<typename InputType, typename OutputType>
ConcatType<InputType, OutputType>::~ConcatType()
{
// Clear memory.
for (size_t i = 0; i < this->network.size(); ++i)
delete this->network[i];
}
template<typename InputType, typename OutputType>
void ConcatType<InputType, OutputType>::Forward(
const InputType& input, OutputType& output)
{
this->InitializeForwardPassMemory();
// Pass the input through all the layers in the network.
for (size_t i = 0; i < this->network.size(); ++i)
{
this->network[i]->Forward(input, this->layerOutputs[i]);
}
// Now concatenate the outputs along the correct axis.
// We can actually use Armadillo to do this for us---we will treat the axis of
// interest as "columns", any axes that come before the axis of interest as
// 'flattened slices', and any axes that come after the axis of interest as
// 'flattened rows'. As a result, we will only have to do join_cols() to
// produce the right result.
//
// Note that we will have one "extra" axis in addition to
// this->outputDimensions.size(); that is the batch size (represented as the
// number of columns in `input`).
size_t slices = (axis == 0) ? input.n_cols :
std::accumulate(this->outputDimensions.begin(),
this->outputDimensions.begin() + axis, 0) + input.n_cols;
size_t rows = (axis == this->outputDimensions.size() - 1) ? 1 :
std::accumulate(this->outputDimensions.begin() + axis + 1,
this->outputDimensions.end(), 0);
std::vector<arma::Cube<typename OutputType::elem_type>> layerOutputAliases;
for (size_t i = 0; i < this->layerOutputs.size(); ++i)
{
layerOutputAliases.emplace_back(arma::Cube<typename OutputType::elem_type>(
this->layerOutputs[i].memptr(), rows,
this->network[i]->OutputDimensions()[axis], slices, false, true);
}
arma::Cube<typename OutputType::elem_type> output(output.memptr(), rows,
this->outputDimensions[axis], slices, false, true);
// Now get the columns from each output.
size_t startCol = 0;
for (size_t i = 0; i < layerOutputAliases.size(); ++i)
{
const size_t cols = layerOutputAliases[i].n_cols;
output.cols(startCol, startCol + cols - 1) = layerOutputAliases[i];
startCol += cols;
}
}
template<typename InputType, typename OutputType>
void ConcatType<InputType, OutputType>::Backward(
const InputType& /* input */, const OutputType& gy, OutputType& g)
{
this->InitializeBackwardPassMemory();
// Just like the forward pass, we can treat our inputs as a cube, but here we
// have to distribute the correct parts of `gy` to the layers.
size_t slices = (axis == 0) ? gy.n_cols :
std::accumulate(this->outputDimensions.begin(),
this->outputDimensions.begin() + axis, 0) + gy.n_cols;
size_t rows = (axis == this->outputDimensions.size() - 1) ? 1 :
std::accumulate(this->outputDimensions.begin() + axis + 1,
this->outputDimensions.end(), 0);
arma::Cube<typename OutputType::elem_type> gyTmp(gy.memptr(), rows,
this->outputDimensions[axis], slices, false, true);
size_t startCol = 0;
for (size_t i = 0; i < this->network.size(); ++i)
{
const size_t cols = this->network[i]->OutputDimensions()[axis];
// TODO: is delta size correct?
// TODO: no copy!
OutputType delta = gyTmp.cols(startCol, startCol + cols - 1);
// TODO: consider batch size correctly
delta.reshape( ... );
this->network[i]->Backward(this->layerOutputs[i], delta,
this->layerDeltas[i]);
startCol += cols;
}
g = this->layerDeltas[0];
for (size_t i = 1; i < this->network.size(); ++i)
{
g += this->layerDeltas[i];
}
}
template<typename InputType, typename OutputType>
void ConcatType<InputType, OutputType>::Backward(
const InputType& /* input */,
const OutputType& gy,
OutputType& g,
const size_t index)
{
// We only intend to perform a backward pass on one layer.
// Thus, we need to extract the parts of gy that correspond to the desired
// layer (specified by `index`).
size_t slices = (axis == 0) ? gy.n_cols :
std::accumulate(this->outputDimensions.begin(),
this->outputDimensions.begin() + axis, 0) + gy.n_cols;
size_t rows = (axis == this->outputDimensions.size() - 1) ? 1 :
std::accumulate(this->outputDimensions.begin() + axis + 1,
this->outputDimensions.end(), 0);
arma::Cube<typename OutputType::elem_type> gyTmp(gy.memptr(), rows,
this->outputDimensions[axis], slices, false, true);
size_t startCol = 0;
for (size_t i = 0; i < index; ++i)
{
startCol += this->network[i]->OutputDimensions()[axis];
}
// TODO: no copy!
const size_t cols = this->network[index]->OutputDimensions()[axis];
OutputType delta = gyTmp.cols(startCol, startCol + cols - 1);
delta.reshape( ... );
this->network[index]->Backward(this->layerOutputs[index], delta, g);
}
template<typename InputType, typename OutputType>
void ConcatType<InputType, OutputType>::Gradient(
const InputType& input,
const OutputType& error,
OutputType& gradient)
{
// Just like the forward pass, we can treat our inputs as a cube, but here we
// have to distribute the correct parts of `gy` to the layers.
size_t slices = (axis == 0) ? input.n_cols :
std::accumulate(this->outputDimensions.begin(),
this->outputDimensions.begin() + axis, 0) + input.n_cols;
size_t rows = (axis == this->outputDimensions.size() - 1) ? 1 :
std::accumulate(this->outputDimensions.begin() + axis + 1,
this->outputDimensions.end(), 0);
arma::Cube<typename OutputType::elem_type> errorTmp(error.memptr(), rows,
this->outputDimensions[axis], slices, false, true);
size_t startCol = 0;
size_t startParam = 0;
for (size_t i = 0; i < this->network.size(); ++i)
{
const size_t cols = this->network[i]->OutputDimensions()[axis];
const size_t params = this->network[i]->WeightSize();
OutputType err = errorTmp.cols(startCol, startCol + cols - 1);
err.reshape(input.n_cols, err.n_elem / input.n_cols);
// TODO: what about layerGradients?
OutputType gradientAlias(gradient.colptr(startParam, 1, params, false,
true);
this->network[i]->Gradient(input, err, gradientAlias);
startCol += cols;
startParam += params;
}
}
// TODO: adapt
template<typename InputType, typename OutputType>
void ConcatType<InputType, OutputType>::Gradient(
const InputType& input,
const OutputType& error,
OutputType& gradient,
const size_t index)
{
// Just like the forward pass, we can treat our inputs as a cube, but here we
// have to distribute the correct parts of `gy` to the layers.
size_t slices = (axis == 0) ? input.n_cols :
std::accumulate(this->outputDimensions.begin(),
this->outputDimensions.begin() + axis, 0) + input.n_cols;
size_t rows = (axis == this->outputDimensions.size() - 1) ? 1 :
std::accumulate(this->outputDimensions.begin() + axis + 1,
this->outputDimensions.end(), 0);
arma::Cube<typename OutputType::elem_type> errorTmp(error.memptr(), rows,
this->outputDimensions[axis], slices, false, true);
size_t startCol = 0;
size_t startParam = 0;
for (size_t i = 0; i < index; ++i)
{
startCol += this->network[i]->OutputDimensions()[axis];
startParam += this->network[i]->WeightSize();
}
const size_t cols = this->network[index]->OutputDimensions()[axis];
const size_t params = this->network[index]->WeightSize();
// TODO: no copy!
OutputType err = errorTmp.cols(startCol, startCol + cols - 1);
err.reshape(input.n_cols, err.n_elem / input.n_cols);
OutputType gradientAlias(gradient.memptr(), 1, params, false, true);
this->network[index]->Gradient(input, err, gradientAlias);
}
template<typename InputType, typename OutputType>
template<typename Archive>
void ConcatType<InputType, OutputType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(cereal::base_class<MultiLayer<InputType, OutputType>>(this));
ar(CEREAL_NVP(axis));
ar(CEREAL_NVP(useAxis));
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/concat_performance.hpp → src/mlpack/methods/ann/layer/not_adapted/concat_performance.hpp
+15 -30
View File
@@ -24,17 +24,17 @@ namespace ann /** Artificial Neural Network. */ {
* feed-forward fully connected network container which plugs performance layers
* together.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* @tparam InputType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* @tparam OutputType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
*/
template <
typename OutputLayerType = NegativeLogLikelihood<>,
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
typename InputType = arma::mat,
typename OutputType = arma::mat
>
class ConcatPerformance
class ConcatPerformance : public Layer<InputType, OutputType>
{
public:
/**
@@ -43,8 +43,7 @@ class ConcatPerformance
* @param inSize The number of inputs.
* @param outputLayer Output layer used to evaluate the network.
*/
ConcatPerformance(const size_t inSize = 0,
OutputLayerType&& outputLayer = OutputLayerType());
ConcatPerformance(OutputLayerType&& outputLayer = OutputLayerType());
/*
* Computes the Negative log likelihood.
@@ -52,8 +51,7 @@ class ConcatPerformance
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename eT>
double Forward(const arma::Mat<eT>& input, arma::Mat<eT>& target);
void Forward(const InputType& input, OutputType& target);
/**
* Ordinary feed backward pass of a neural network. The negative log
@@ -66,42 +64,29 @@ class ConcatPerformance
* between 1 and the number of classes.
* @param output The calculated error.
*/
template<typename eT>
void Backward(const arma::Mat<eT>& input,
const arma::Mat<eT>& target,
arma::Mat<eT>& output);
void Backward(const InputType& input,
const OutputType& target,
OutputType& output);
//! Get the output parameter.
OutputDataType& OutputParameter() const { return outputParameter; }
OutputType& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
OutputType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType& Delta() const { return delta; }
OutputType& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the number of inputs.
size_t InSize() const { return inSize; }
OutputType& Delta() { return delta; }
/**
* Serialize the layer
* Serialize the layer.
*/
template<typename Archive>
void serialize(Archive& /* ar */, const uint32_t /* version */);
private:
//! Locally-stored number of inputs.
size_t inSize;
//! Instantiated outputlayer used to evaluate the network.
OutputLayerType outputLayer;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
}; // class ConcatPerformance
} // namespace ann
src/mlpack/methods/ann/layer/concat_performance_impl.hpp → src/mlpack/methods/ann/layer/not_adapted/concat_performance_impl.hpp
+35 -34
View File
@@ -20,15 +20,14 @@ namespace ann /** Artificial Neural Network. */ {
template<
typename OutputLayerType,
typename InputDataType,
typename OutputDataType
typename InputType,
typename OutputType
>
ConcatPerformance<
OutputLayerType,
InputDataType,
OutputDataType
>::ConcatPerformance(const size_t inSize, OutputLayerType&& outputLayer) :
inSize(inSize),
InputType,
OutputType
>::ConcatPerformance(OutputLayerType&& outputLayer) :
outputLayer(std::move(outputLayer))
{
// Nothing to do here.
@@ -36,51 +35,51 @@ ConcatPerformance<
template<
typename OutputLayerType,
typename InputDataType,
typename OutputDataType
typename InputType,
typename OutputType
>
template<typename eT>
double ConcatPerformance<
void ConcatPerformance<
OutputLayerType,
InputDataType,
OutputDataType
>::Forward(const arma::Mat<eT>& input, arma::Mat<eT>& target)
InputType,
OutputType
>::Forward(const InputType& input, OutputType& target)
{
const size_t elements = input.n_elem / inSize;
const size_t elements = input.n_elem / inputDimensions[0];
double output = 0;
for (size_t i = 0; i < input.n_elem; i+= elements)
for (size_t i = 0; i < input.n_elem; i += elements)
{
arma::mat subInput = input.submat(i, 0, i + elements - 1, 0);
InputType subInput = input.submat(i, 0, i + elements - 1, 0);
output += outputLayer.Forward(subInput, target);
}
return output;
// TODO: what to do with output?
//return output;
return;
}
template<
typename OutputLayerType,
typename InputDataType,
typename OutputDataType
typename InputType,
typename OutputType
>
template<typename eT>
void ConcatPerformance<
OutputLayerType,
InputDataType,
OutputDataType
InputType,
OutputType
>::Backward(
const arma::Mat<eT>& input,
const arma::Mat<eT>& target,
arma::Mat<eT>& output)
const InputType& input,
const OutputType& target,
OutputType& output)
{
const size_t elements = input.n_elem / inSize;
const size_t elements = input.n_elem / inputDimensions[0];
arma::mat subInput = input.submat(0, 0, elements - 1, 0);
arma::mat subOutput;
InputType subInput = input.submat(0, 0, elements - 1, 0);
OutputType subOutput;
outputLayer.Backward(subInput, target, subOutput);
output = arma::zeros(subOutput.n_elem, inSize);
output = arma::zeros(subOutput.n_elem, inputDimensions[0]);
output.col(0) = subOutput;
for (size_t i = elements, j = 0; i < input.n_elem; i+= elements, ++j)
@@ -94,17 +93,19 @@ void ConcatPerformance<
template<
typename OutputLayerType,
typename InputDataType,
typename OutputDataType
typename InputType,
typename OutputType
>
template<typename Archive>
void ConcatPerformance<
OutputLayerType,
InputDataType,
OutputDataType
InputType,
OutputType
>::serialize(Archive& ar, const uint32_t /* version */)
{
ar(CEREAL_NVP(inSize));
ar(cereal::base_class<Layer<InputType, OutputType>>(this));
ar(CEREAL_NVP(outputLayer));
}
} // namespace ann
src/mlpack/methods/ann/layer/constant.hpp → src/mlpack/methods/ann/layer/not_adapted/constant.hpp
+40 -42
View File
@@ -15,6 +15,8 @@
#include <mlpack/prereqs.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -22,18 +24,21 @@ namespace ann /** Artificial Neural Network. */ {
* Implementation of the constant layer. The constant layer outputs a given
* constant value given any input value.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam InputType The type of the layer's inputs. The layer automatically
* cast inputs to this type (Default: arma::mat).
* @tparam OutputType The type of the computation which also causes the output
* to also be in this type. The type also allows the computation and weight
* type to differ from the input type (Default: arma::mat).
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class Constant
template<typename InputType = arma::mat, typename OutputType = arma::mat>
class ConstantType : public Layer<InputType, OutputType>
{
public:
/**
* Create an empty Constant layer.
*/
ConstantType();
/**
* Create the Constant object that outputs a given constant scalar value
* given any input value.
@@ -41,7 +46,19 @@ class Constant
* @param outSize The number of output units.
* @param scalar The constant value used to create the constant output.
*/
Constant(const size_t outSize = 0, const double scalar = 0.0);
ConstantType(const size_t outSize, const double scalar = 0);
//! Copy another ConstantType.
ConstantType(const ConstantType& layer);
//! Take ownership of another ConstantType.
ConstantType(ConstantType&& layer);
//! Copy another ConstantType.
ConstantType& operator=(const ConstantType& layer);
//! Take ownership of another ConstantType.
ConstantType& operator=(ConstantType&& layer);
//! Clone the ConstantType object. This handles polymorphism correctly.
ConstantType* Clone() const { return new ConstantType(*this); }
/**
* Ordinary feed forward pass of a neural network. The forward pass fills the
@@ -50,7 +67,6 @@ class Constant
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename InputType, typename OutputType>
void Forward(const InputType& input, OutputType& output);
/**
@@ -61,52 +77,34 @@ class Constant
* @param * (gy) The backpropagated error.
* @param g The calculated gradient.
*/
template<typename DataType>
void Backward(const DataType& /* input */,
const DataType& /* gy */,
DataType& g);
//! Get the output parameter.
OutputDataType& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
void Backward(const InputType& /* input */,
const OutputType& /* gy */,
OutputType& g);
//! Get the output size.
size_t OutSize() const { return outSize; }
//! Get the size of the weights.
size_t WeightSize() const
const std::vector<size_t>& OutputDimensions() const
{
return 0;
std::vector<size_t> result(this->inputDimensions.size(), 0);
result[0] = outSize;
return result;
}
/**
* Serialize the layer.
*/
//! Serialize the layer.
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Locally-stored number of input units.
size_t inSize;
//! Locally-stored number of output units.
size_t outSize;
//! Locally-stored constant output matrix.
OutputDataType constantOutput;
OutputType constantOutput;
}; // class ConstantType
//! Locally-stored delta object.
OutputDataType delta;
// Convenience typedefs.
//! Locally-stored output parameter object.
OutputDataType outputParameter;
}; // class ConstantLayer
// Standard HardShrink layer.
typedef ConstantType<arma::mat, arma::mat> Constant;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/not_adapted/constant_impl.hpp
+118
View File
@@ -0,0 +1,118 @@
/**
* @file methods/ann/layer/constant_impl.hpp
* @author Marcus Edel
*
* Implementation of the Constant class, which outputs a constant value given
* any input.
*
* mlpack is free software; you may redistribute it and/or modify it under the
* terms of the 3-clause BSD license. You should have received a copy of the
* 3-clause BSD license along with mlpack. If not, see
* http://www.opensource.org/licenses/BSD-3-Clause for more information.
*/
#ifndef MLPACK_METHODS_ANN_LAYER_CONSTANT_IMPL_HPP
#define MLPACK_METHODS_ANN_LAYER_CONSTANT_IMPL_HPP
// In case it hasn't yet been included.
#include "constant.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
template<typename InputType, typename OutputType>
ConstantType<InputType, OutputType>::ConstantType() :
outSize(0)
{
// Nothing to do.
}
template<typename InputType, typename OutputType>
ConstantType<InputType, OutputType>::ConstantType(
const size_t outSize,
const double scalar) :
outSize(outSize)
{
constantOutput = OutputType(outSize, 1);
constantOutput.fill(scalar);
}
template<typename InputType, typename OutputType>
ConstantType<InputType, OutputType>::ConstantType(
const ConstantType<InputType, OutputType>& other) :
outSize(other.outSize),
constantOutput(other.constantOutput)
{
// Nothing else to do.
}
template<typename InputType, typename OutputType>
ConstantType<InputType, OutputType>::ConstantType(
ConstantType<InputType, OutputType>&& other) :
outSize(other.outSize),
constantOutput(std::move(other.constantOutput))
{
other.outSize = 1;
other.constantOutput = OutputType(other.outSize, 1);
}
template<typename InputType, typename OutputType>
ConstantType<InputType, OutputType>&
ConstantType<InputType, OutputType>::operator=(
const ConstantType<InputType, OutputType>& other)
{
if (this != &other)
{
outSize = other.outSize;
constantOutput = other.constantOutput;
}
return *this;
}
template<typename InputType, typename OutputType>
ConstantType<InputType, OutputType>&
ConstantType<InputType, OutputType>::operator=(
ConstantType<InputType, OutputType>&& other)
{
if (this != *other)
{
outSize = other.outSize;
constantOutput = std::move(other.constantOutput);
other.outSize = 1;
other.constantOutput = OutputType(other.outSize, 1);
}
return *this;
}
template<typename InputType, typename OutputType>
void ConstantType<InputType, OutputType>::Forward(
const InputType& input, OutputType& output)
{
output = constantOutput;
}
template<typename InputType, typename OutputType>
void ConstantType<InputType, OutputType>::Backward(
const InputType& /* input */, const OutputType& /* gy */, OutputType& g)
{
g.zeros();
}
template<typename InputType, typename OutputType>
template<typename Archive>
void ConstantType<InputType, OutputType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(cereal::base_class<Layer<InputType, OutputType>>(this));
ar(CEREAL_NVP(constantOutput));
if (Archive::is_loading::value)
outSize = constantOutput.n_elem;
}
} // namespace ann
} // namespace mlpack
#endif
src/mlpack/methods/ann/layer/elu.hpp → src/mlpack/methods/ann/layer/not_adapted/elu.hpp
+27 -49
View File
@@ -6,9 +6,9 @@
* Definition of the ELU activation function as described by Djork-Arne Clevert,
* Thomas Unterthiner and Sepp Hochreiter.
*
* Definition of the SELU function as introduced by
* Klambauer et. al. in Self Neural Networks. The SELU activation
* function keeps the mean and variance of the input invariant.
* Definition of the SELU function as introduced by Klambauer et. al. in Self
* Neural Networks. The SELU activation function keeps the mean and variance of
* the input invariant.
*
* In short, SELU = lambda * ELU, with 'alpha' and 'lambda' fixed for
* normalized inputs.
@@ -26,6 +26,8 @@
#include <mlpack/prereqs.hpp>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -61,7 +63,6 @@ namespace ann /** Artificial Neural Network. */ {
* }
* @endcode
*
*
* The SELU activation function is defined by
*
* @f{eqnarray*}{
@@ -92,23 +93,19 @@ namespace ann /** Artificial Neural Network. */ {
* }
* @endcode
*
* In the deterministic mode, there is no computation of the derivative.
* In testing mode, there is no computation of the derivative.
*
* @note During training deterministic should be set to false and during
* testing/inference deterministic should be set to true.
* @note Make sure to use SELU activation function with normalized inputs and
* weights initialized with Lecun Normal Initialization.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam InputType The type of the layer's inputs. The layer automatically
* cast inputs to this type (Default: arma::mat).
* @tparam OutputType The type of the computation which also causes the output
* to also be in this type. The type also allows the computation and weight
* type to differ from the input type (Default: arma::mat).
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
>
class ELU
template <typename InputType = arma::mat, typename OutputType = arma::mat>
class ELUType : public Layer<InputType, OutputType>
{
public:
/**
@@ -116,7 +113,7 @@ class ELU
*
* NOTE: Use this constructor for SELU activation function.
*/
ELU();
ELUType();
/**
* Create the ELU object using the specified parameter. The non zero
@@ -126,8 +123,10 @@ class ELU
* @note Use this constructor for ELU activation function.
* @param alpha Scale parameter for the negative factor.
*/
ELU(const double alpha);
ELUType(const double alpha);
//! Clone the ELUType object. This handles polymorphism correctly.
ELUType* Clone() const { return new ELUType(*this); }
/**
* Ordinary feed forward pass of a neural network, evaluating the function
* f(x) by propagating the activity forward through f.
@@ -135,7 +134,6 @@ class ELU
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename InputType, typename OutputType>
void Forward(const InputType& input, OutputType& output);
/**
@@ -147,29 +145,13 @@ class ELU
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename DataType>
void Backward(const DataType& input, const DataType& gy, DataType& g);
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
void Backward(const InputType& input, const OutputType& gy, OutputType& g);
//! Get the non zero gradient.
double const& Alpha() const { return alpha; }
//! Modify the non zero gradient.
double& Alpha() { return alpha; }
//! Get the value of deterministic parameter.
bool Deterministic() const { return deterministic; }
//! Modify the value of deterministic parameter.
bool& Deterministic() { return deterministic; }
//! Get the lambda parameter.
double const& Lambda() const { return lambda; }
@@ -180,31 +162,27 @@ class ELU
void serialize(Archive& ar, const uint32_t /* version */);
private:
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
//! Locally stored first derivative of the activation function.
arma::mat derivative;
OutputType derivative;
//! ELU Hyperparameter (0 < alpha)
//! SELU parameter fixed to 1.6732632423543774 for normalized inputs.
double alpha;
//! Lambda Parameter used for multiplication of ELU function.
//! Lambda parameter used for multiplication of ELU function.
//! For ELU activation function, lambda = 1.
//! For SELU activation function, lambda = 1.0507009873554802 for normalized
//! inputs.
double lambda;
}; // class ELUType
//! If true the derivative computation is disabled, see notes above.
bool deterministic;
}; // class ELU
// Convenience typedefs.
// Template alias for SELU using ELU class.
using SELU = ELU<arma::mat, arma::mat>;
// Standard flexible ReLU layer.
typedef ELUType<arma::mat, arma::mat> ELU;
// Standard ELU layer.
typedef ELUType<arma::mat, arma::mat> SELU;
} // namespace ann
} // namespace mlpack
src/mlpack/methods/ann/layer/elu_impl.hpp → src/mlpack/methods/ann/layer/not_adapted/elu_impl.hpp
+35 -42
View File
@@ -26,66 +26,59 @@ namespace ann /** Artificial Neural Network. */ {
// This constructor is called for SELU activation function. The values of
// alpha and lambda are constant for normalized inputs.
template<typename InputDataType, typename OutputDataType>
ELU<InputDataType, OutputDataType>::ELU() :
alpha(1.6732632423543774),
lambda(1.0507009873554802),
deterministic(false)
{
// Nothing to do here.
}
// This constructor is called for ELU activation function. The value of lambda
// is fixed and equal to 1. 'alpha' is a hyperparameter.
template<typename InputDataType, typename OutputDataType>
ELU<InputDataType, OutputDataType>::ELU(const double alpha) :
alpha(alpha),
lambda(1),
deterministic(false)
{
// Nothing to do here.
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
void ELU<InputDataType, OutputDataType>::Forward(
ELUType<InputType, OutputType>::ELUType() :
alpha(1.6732632423543774),
lambda(1.0507009873554802)
{
// Nothing to do here.
}
// This constructor is called for ELU activation function. The value of lambda
// is fixed and equal to 1. 'alpha' is a hyperparameter.
template<typename InputType, typename OutputType>
ELUType<InputType, OutputType>::ELUType(const double alpha) :
alpha(alpha),
lambda(1)
{
// Nothing to do here.
}
template<typename InputType, typename OutputType>
void ELUType<InputType, OutputType>::Forward(
const InputType& input, OutputType& output)
{
output = arma::ones<OutputDataType>(arma::size(input));
output.ones();
for (size_t i = 0; i < input.n_elem; ++i)
{
if (input(i) < DBL_MAX)
{
output(i) = (input(i) > 0) ? lambda * input(i) : lambda *
alpha * (std::exp(input(i)) - 1);
output(i) = (input(i) > 0) ? lambda * input(i) : lambda * alpha *
(std::exp(input(i)) - 1);
}
}
if (!deterministic)
{
derivative.set_size(arma::size(input));
for (size_t i = 0; i < input.n_elem; ++i)
{
derivative(i) = (input(i) > 0) ? lambda : output(i) +
lambda * alpha;
}
}
if (!deterministic)
{
for (size_t i = 0; i < input.n_elem; ++i)
derivative(i) = (input(i) > 0) ? lambda : output(i) + lambda * alpha;
}
}
template<typename InputDataType, typename OutputDataType>
template<typename DataType>
void ELU<InputDataType, OutputDataType>::Backward(
const DataType& /* input */, const DataType& gy, DataType& g)
template<typename InputType, typename OutputType>
void ELUType<InputType, OutputType>::Backward(
const InputType& /* input */, const OutputType& gy, OutputType& g)
{
g = gy % derivative;
}
template<typename InputDataType, typename OutputDataType>
template<typename InputType, typename OutputType>
template<typename Archive>
void ELU<InputDataType, OutputDataType>::serialize(
Archive& ar,
const uint32_t /* version */)
void ELUType<InputType, OutputType>::serialize(
Archive& ar, const uint32_t /* version */)
{
ar(cereal::base_class<Layer<InputType, OutputType>>(this));
ar(CEREAL_NVP(alpha));
ar(CEREAL_NVP(lambda));
}
src/mlpack/methods/ann/layer/fast_lstm.hpp → src/mlpack/methods/ann/layer/not_adapted/fast_lstm.hpp
+57 -76
View File
@@ -1,3 +1,4 @@
// Temporarily drop.
/**
* @file methods/ann/layer/fast_lstm.hpp
* @author Marcus Edel
@@ -15,6 +16,7 @@
#include <mlpack/prereqs.hpp>
#include <limits>
#include "layer.hpp"
namespace mlpack {
namespace ann /** Artificial Neural Network. */ {
@@ -54,36 +56,36 @@ namespace ann /** Artificial Neural Network. */ {
*
* \see LSTM for a standard implementation of the LSTM layer.
*
* @tparam InputDataType Type of the input data (arma::colvec, arma::mat,
* @tparam InputType Type of the input data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
* @tparam OutputDataType Type of the output data (arma::colvec, arma::mat,
* @tparam OutputType Type of the output data (arma::colvec, arma::mat,
* arma::sp_mat or arma::cube).
*/
template <
typename InputDataType = arma::mat,
typename OutputDataType = arma::mat
typename InputType = arma::mat,
typename OutputType = arma::mat
>
class FastLSTM
class FastLSTMType : public Layer<InputType, OutputType>
{
public:
// Convenience typedefs.
typedef typename InputDataType::elem_type InputElemType;
typedef typename OutputDataType::elem_type ElemType;
typedef typename InputType::elem_type InputET;
typedef typename OutputType::elem_type OutputET;
//! Create the Fast LSTM object.
FastLSTM();
//! Create the FastLSTMType object.
FastLSTMType();
//! Copy Constructor
FastLSTM(const FastLSTM& layer);
FastLSTMType(const FastLSTMType& layer);
//! Move Constructor
FastLSTM(FastLSTM&& layer);
FastLSTMType(FastLSTMType&& layer);
//! Copy assignment operator
FastLSTM& operator=(const FastLSTM& layer);
FastLSTMType& operator=(const FastLSTMType& layer);
//! Move assignment operator
FastLSTM& operator=(FastLSTM&& layer);
FastLSTMType& operator=(FastLSTMType&& layer);
/**
* Create the Fast LSTM layer object using the specified parameters.
@@ -92,9 +94,12 @@ class FastLSTM
* @param outSize The number of output units.
* @param rho Maximum number of steps to backpropagate through time (BPTT).
*/
FastLSTM(const size_t inSize,
const size_t outSize,
const size_t rho = std::numeric_limits<size_t>::max());
FastLSTMType(const size_t inSize,
const size_t outSize,
const size_t rho = std::numeric_limits<size_t>::max());
//! Clone the FastLSTMType object. This handles polymorphism correctly.
FastLSTMType* Clone() const { return new FastLSTMType(*this); }
/**
* Ordinary feed forward pass of a neural network, evaluating the function
@@ -103,7 +108,6 @@ class FastLSTM
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
template<typename InputType, typename OutputType>
void Forward(const InputType& input, OutputType& output);
/**
@@ -115,17 +119,16 @@ class FastLSTM
* @param gy The backpropagated error.
* @param g The calculated gradient.
*/
template<typename InputType, typename ErrorType, typename GradientType>
void Backward(const InputType& input,
const ErrorType& gy,
GradientType& g);
const OutputType& gy,
OutputType& g);
/*
/**
* Reset the layer parameter.
*/
void Reset();
/*
/**
* Resets the cell to accept a new input. This breaks the BPTT chain starts a
* new one.
*
@@ -133,17 +136,16 @@ class FastLSTM
*/
void ResetCell(const size_t size);
/*
/**
* Calculate the gradient using the output delta and the input activation.
*
* @param input The input parameter used for calculating the gradient.
* @param error The calculated error.
* @param gradient The calculated gradient.
*/
template<typename InputType, typename ErrorType, typename GradientType>
void Gradient(const InputType& input,
const ErrorType& error,
GradientType& gradient);
const OutputType& error,
OutputType& gradient);
//! Get the maximum number of steps to backpropagate through time (BPTT).
size_t Rho() const { return rho; }
@@ -151,24 +153,9 @@ class FastLSTM
size_t& Rho() { return rho; }
//! Get the parameters.
OutputDataType const& Parameters() const { return weights; }
OutputType const& Parameters() const { return weights; }
//! Modify the parameters.
OutputDataType& Parameters() { return weights; }
//! Get the output parameter.
OutputDataType const& OutputParameter() const { return outputParameter; }
//! Modify the output parameter.
OutputDataType& OutputParameter() { return outputParameter; }
//! Get the delta.
OutputDataType const& Delta() const { return delta; }
//! Modify the delta.
OutputDataType& Delta() { return delta; }
//! Get the gradient.
OutputDataType const& Gradient() const { return grad; }
//! Modify the gradient.
OutputDataType& Gradient() { return grad; }
OutputType& Parameters() { return weights; }
//! Get the number of input units.
size_t InSize() const { return inSize; }
@@ -182,14 +169,15 @@ class FastLSTM
return 4 * outSize * inSize + 4 * outSize + 4 * outSize * outSize;
}
//! Get the shape of the input.
size_t InputShape() const
const std::vector<size_t> OutputDimensions() const
{
return inSize;
std::vector<size_t> result(inputDimensions.size(), 0);
result[0] = outSize;
return result;
}
/**
* Serialize the layer
* Serialize the layer.
*/
template<typename Archive>
void serialize(Archive& ar, const uint32_t /* version */);
@@ -201,7 +189,6 @@ class FastLSTM
* @param input The input data.
* @param sigmoid The matrix to store the sigmoid approximation into.
*/
template<typename InputType, typename OutputType>
void FastSigmoid(const InputType& input, OutputType& sigmoids)
{
for (size_t i = 0; i < input.n_elem; ++i)
@@ -214,10 +201,10 @@ class FastLSTM
* @param data The given data sample for the sigmoid approximation.
* @tparam The sigmoid approximation.
*/
ElemType FastSigmoid(const InputElemType data)
OutputET FastSigmoid(const InputET data)
{
ElemType x = 0.5 * data;
ElemType z;
OutputET x = 0.5 * data;
OutputET z;
if (x >= 0)
{
if (x < 1.7)
@@ -229,7 +216,7 @@ class FastLSTM
}
else
{
ElemType xx = -x;
OutputET xx = -x;
if (xx < 1.7)
z = -(1.5 * xx / (1 + xx));
else if (xx < 3)
@@ -260,10 +247,10 @@ class FastLSTM
size_t gradientStep;
//! Locally-stored weight object.
OutputDataType weights;
OutputType weights;
//! Locally-stored previous output.
OutputDataType prevOutput;
OutputType prevOutput;
//! Locally-stored batch size.
size_t batchSize;
@@ -276,56 +263,50 @@ class FastLSTM
size_t gradientStepIdx;
//! Locally-stored cell activation error.
OutputDataType cellActivationError;
//! Locally-stored delta object.
OutputDataType delta;
//! Locally-stored gradient object.
OutputDataType grad;
OutputType cellActivationError;
//! Locally-stored output parameter object.
OutputDataType outputParameter;
OutputType outputParameter;
//! Weights between the output and gate.
OutputDataType output2GateWeight;
OutputType output2GateWeight;
//! Weights between the input and gate.
OutputDataType input2GateWeight;
OutputType input2GateWeight;
//! Bias between the input and gate.
OutputDataType input2GateBias;
OutputType input2GateBias;
//! Locally-stored gate parameter.
OutputDataType gate;
OutputType gate;
//! Locally-stored gate activation.
OutputDataType gateActivation;
OutputType gateActivation;
//! Locally-stored state activation.
OutputDataType stateActivation;
OutputType stateActivation;
//! Locally-stored cell parameter.
OutputDataType cell;
OutputType cell;
//! Locally-stored cell activation error.
OutputDataType cellActivation;
OutputType cellActivation;
//! Locally-stored foget gate error.
OutputDataType forgetGateError;
OutputType forgetGateError;
//! Locally-stored previous error.
OutputDataType prevError;
//! Locally-stored output parameters.
OutputDataType outParameter;
OutputType prevError;
//! Locally-stored current rho size.
size_t rhoSize;
//! Current backpropagate through time steps.
size_t bpttSteps;
}; // class FastLSTM
}; // class FastLSTMType.
// Standard FastLSTM layer.
typedef FastLSTMType<arma::mat, arma::mat> FastLSTM;
} // namespace ann
} // namespace mlpack

Some files were not shown because too many files have changed in this diff Show More